Post processing toon effect

Posted By: Ganderoleg

Post processing toon effect - 01/20/11 22:15

Hello,

I was just wandering if anyone knows of a good (or at least functional) post processing toon effect? ..does something like this even exists? I guess that GS Sobel is somewhat close to the effect but it's not really toon:(
Posted By: Quad

Re: Post processing toon effect - 01/20/11 22:31

http://www.opserver.de/wiki/index.php/Toonshading
http://www.opserver.de/wiki/index.php/Toonshading_with_Outline
Posted By: Ganderoleg

Re: Post processing toon effect - 01/20/11 22:54

I've tried 'Toonshading with Outline' & it seems to be working great:)
Thanx for links, I never seem to remember to check out GS wiki when in need of code:)
Posted By: Ganderoleg

Re: Post processing toon effect - 01/22/11 13:07

..well I have a big problem with the outlining Toon ppe from wiki, it doesn't seem to be compatible with fsaa.dll that I use for smoothing out the edges:( No matter what I do (activate Toon ppe through Shade-C or directlly through camera.stage) the edges are totally unaffected & anti-aliasing is zero.

Does anyone has any idea on why this is happening or how can it be fixed? It's not specific to one project, it's the same thing no matter where I try to use it.
Posted By: Rondidon

Re: Post processing toon effect - 01/22/11 13:36

Same request here. I would love to have HDR (Shade-C)+FSAA for example.
Posted By: Ganderoleg

Re: Post processing toon effect - 01/22/11 14:00

@Rondidon
You should try implementing it in, maybe, a different way because I don't have any troubles with combining Shade-C & FSAA, it works fine..



..but this toon ppe is driving me insane, edges just refuse to be ironed-out:(
Posted By: Rondidon

Re: Post processing toon effect - 01/22/11 14:28

Which FSAA ppe do you use? The GameStudio-native one or a plugin / dll? If so, where can I find it? Or is FSAA a Shade-C feature that I missed?

Interesting car on the screenshot laugh Reminds me on Carmageddon or Pod (old 3DFX games). I like your artwork. Your games have a unique style.
Posted By: Ganderoleg

Re: Post processing toon effect - 01/22/11 14:58

Thanx, the car textures & geometry is still in design stage but it shows potential:)

I am using happily for some time now one of the best dll forum contributions of all times: ChrisB's Fullscreen Antialiasing plugin :):) It works in almost 90% of the cases & it's C-Script/Lite-C friendly, I would recommend you try it.

You can download dll here:
http://www.opserver.de/ubb7/ubbthreads.php?ubb=showflat&Number=330830&page=1
Posted By: Ganderoleg

Re: Post processing toon effect - 01/22/11 20:15

Well after some time I decided to go for a completely different approach & instead of ppe use simple toon shader. The reason for this is not only anti-aliasing (it is about 80%:) but also the fact that outline is constantly changing because it's not bounded to object but to camera view.

The problem was that my old files are all in wmb format & I didn't want to transform them to mdl's & do shadows again (too much work:) So I did the following: at the same place as the wmb file I placed the same geometry in mdl format but with one transparent texture & toon shader:) This way I preserved the original file but also added black outlining. It doesn't really have toon appearance but it has a sort of artistic-toon-like thing:) ..and turning the wmp to mdl takes about 8min which is much, much faster then what I would have to do if I was to create completely new shadowmaps:)

Here are some screens, I still haven't decided if I should go with textured or untextured geometry..









..but I will probably go with untextured rooms, they seems to be more visually interesting:)
Posted By: Superku

Re: Post processing toon effect - 01/22/11 20:31

The untextured rooms look fantastic!
Posted By: Ganderoleg

Re: Post processing toon effect - 01/22/11 20:49

Thanx, they do seem to look more unusual:) It is too bad that the toon ppe is not functioning as I expected but I believe that this is maybe even better, it gives a completely unique toon look to the levels:)
Posted By: Roel

Re: Post processing toon effect - 01/22/11 21:14

about the not working fsaa:

the fsaa dll probably works, but the edges aren't postprocessed, as they are calculated afterwards. and there is an edge, or there is not...
There is no blurring between them.
you could overcome this by tweaking the toon shader, or by applying a very small blurring after the toon shader(a radius of... 1, 2 pixels).

sorry for my bad english, but I hope you get my point.
Posted By: Hummel

Re: Post processing toon effect - 01/22/11 23:40

@Ganderoleg: as far as I understand you are using a transparent model-version of your level geometry which got the toon effect added? If so, the transparent model wastes calculation time since the toon shader renders the model twice, once the model itself and another time for the outlines. Means you need only the second pass. Take a look in the shader and comment out the first pass, so you save a lot performance. wink
Posted By: Ganderoleg

Re: Post processing toon effect - 01/23/11 13:54

@Roel
I think you are right, the fsaa works when toon ppe is not active & it would be logical to assume that it also works when toon ppe is active. My programming skills are not really that impressive so I wouldn't know how to tweak the shader in order to fix the edges, but I tried to add blur (both classic & Gaussian) & that definitely didn't produce desired visual effect.

..but the toon shader from GS folder gets anti-aliased (edges are nice & smooth), so I think this is the card to play at the moment. As I mentioned, it seems to be even better solution since outlines don't change during camera movement:)

@Hummel
Yes, I use two rooms, one in wmb & one in mdl format. The wmb room has one texture & shadowmap and the mdl room has transparent texture & toon shader applied.
I modified the fx file as you suggested (I think I did:) so this is what I have now:

Click to reveal..
#include <transform>
#include <fog>
#include <pos>
#include <normal>
#include <tangent>
#include <light>

float4 vecLight;

float4 vecSkill41;
//vecSkill41.x: float facAmbient = 1.0;
//vecSkill41.y: float facEdgeWidth = 1.0;

texture entSkin1; // texture
sampler sBaseTex = sampler_state { Texture = <entSkin1>; MipFilter = Linear; };

struct toonOut
{
float4 Pos: POSITION;
float Fog: FOG;
float4 Color: COLOR;
float4 Ambient:COLOR1;
float2 Tex: TEXCOORD0;
float3 Normal: TEXCOORD1;
float3 Light: TEXCOORD2;
};

toonOut toonVS(
in float4 inPos: POSITION,
in float3 inNormal: NORMAL,
in float2 inTex: TEXCOORD0,
in float3 inTangent: TEXCOORD2)
{
toonOut Out;
Out.Pos = DoTransform(inPos);
Out.Tex = inTex;
Out.Fog = DoFog(inPos);

float facAmbient = vecSkill41.x / 100.;
Out.Ambient = facAmbient*vecLight;

CreateTangents(inNormal,inTangent);
float3 P = DoPos(inPos);

float3 lightVec;
lightVec = normalize(vecLightPos[0].xyz - P);
Out.Color = vecLightColor[0];

Out.Light = DoTangent(lightVec) * 0.5 + 0.5;
Out.Normal = DoTangent(DoPos(inNormal)) * 0.5 + 0.5;

return Out;
}

float4 toonPS(toonOut In): COLOR
{
float4 base = tex2D(sBaseTex,In.Tex);

float intensity = dot(In.Light*2 - 1,In.Normal*2 - 1);

float light;
if (intensity > 0.85 )
light = 1.;
else if (intensity > 0.35 )
light = 0.55;
else
light = 0.05;

return base * (light*In.Color + In.Ambient);
}

technique toon
{
//pass one
//{
// VertexShader = compile vs_1_1 toonVS();
// PixelShader = compile ps_1_4 toonPS();
//}

pass two
{
CULLMODE=CW;
vertexShaderConstant[0] = <matWorldViewProj>;
vertexShaderConstant[16] = <vecSkill41.y/20.>;

VertexShader = asm
{
vs_1_0
dcl_position v0
dcl_normal v3
dcl_texcoord v7
mov r0,v0
mul r1,c16.x,v3
// Scale the normal
add r0.xyz,r0.xyz,r1.xyz
m4x4 oPos,r0,c0
// Transorm position to clip space
mov oD0, c0
mov r0,c0
};
}
}

//technique fallback { pass one { } }

..is there anything else that I can exclude from the fx without damaging it's functionality?
Posted By: Hummel

Re: Post processing toon effect - 01/25/11 15:54

as far I can see it all you need is that part of the effect:
Code:
technique toon
{

pass outlines
{
CULLMODE=CW;
vertexShaderConstant[0] = <matWorldViewProj>;
vertexShaderConstant[16] = <vecSkill41.y/20.>;

VertexShader = asm
{
vs_1_0
dcl_position v0
dcl_normal v3
dcl_texcoord v7
mov r0,v0
mul r1,c16.x,v3
// Scale the normal
add r0.xyz,r0.xyz,r1.xyz
m4x4 oPos,r0,c0
// Transorm position to clip space
mov oD0, c0
mov r0,c0
};
}
}



PS: what exactly do you mean with "the outlines do not change during camera movement"? Do you refere to the inconsistence of pp-outlines or the fact that geometry based outlines thickness is realtive to the distance of the viewer?
Posted By: Ganderoleg

Re: Post processing toon effect - 01/26/11 20:58

You are right, these lines of code are fully sufficient for shader to work in a manner I wanted:) As for changing of the outlines, I was only thinking about inconsistency of the pp-outlines.





.................
But, unfortunately, I have a strange problem occurring with the model when using WED versions higher then 7.50 (7.84). For some reason the outlines are changing colors when camera is changing the pan, going from black to red, green etc...

I don't need to use versions other then 7.50 for compiling the game but I would like to have a widescreen option which is possible only in higher versions:(

I have uploaded a small zip (4mb) with two mdl files, first fully textured with baked shadows & second mono-textured with modified toon shader applied. When running the second model there should be radical color changes when model is being rotated:(

First fully textured model with baked shadows...


Second mono-textured model with modified toon shader applied...


Outlines with changed colors...


If someone could find time to download this zip & maybe try to see how this could be solved I would be very grateful, it's really beyond what I can fix.

The download link is:
http://www.filefront.com/17854051/ToonFxTest.zip

Thanx in advance to everyone who will download & check out the zip:)

© 2024 lite-C Forums