For one of my games I need to have a toon shader that works in both black and white and color. This is what I created for the black and white one:

here's the script:
Code:

---

 material mat_bw 
{	
	effect = "		
	sampler2D g_samSrcColor;

	float4 MyShader( float2 Tex : TEXCOORD0 ) : COLOR0
	{
		float4 Color;
		int Color_all;
		Color = tex2D( g_samSrcColor, Tex.xy);
		Color.rgb = (Color.r + Color.g + Color.b)/3;
		Color_all = (Color.r/.167);
		Color.rgb = Color_all * .167;
		return Color;
	}


	technique PostProcess
	{
		pass p1
		{
			VertexShader = null;
			PixelShader = compile ps_2_0 MyShader();
		}

	}
	";
} 

---

Now, the problem comes when I try to use the same kind of thing with color. I can't find a formula that would force the saturation to be a certain level. Any suggestions?

By the way, here's what I have for my color toon shader:

And the script:
Code:

---

 material mat_rgb 
{	
	effect = "		
	sampler2D g_samSrcColor;

	float4 MyShader( float2 Tex : TEXCOORD0 ) : COLOR0
	{
		float4 Color;
		int Color_all;
		Color = tex2D( g_samSrcColor, Tex.xy);
		Color_all = ((Color.r + Color.g + Color.b)/3)/.167;
		if(Color.r > Color.b && Color.g > Color.b)
		{
			Color.b = 3*((Color_all*.167)+ .084)-Color.r-Color.g;
		}
		if(Color.b > Color.r && Color.g > Color.r)
		{
			Color.r = 3*((Color_all*.167)+ .084)-Color.b-Color.g;
		}
		if(Color.r > Color.g && Color.b > Color.g)
		{
			Color.g = 3*((Color_all*.167)+ .084)-Color.r-Color.b;
		}
		if(Color.r == Color.b && Color.r == Color.g)
		{
			Color.rgb = Color_all * .167;
		}
		return Color;
	}


	technique PostProcess
	{
		pass p1
		{
			VertexShader = null;
			PixelShader = compile ps_2_0 MyShader();
		}

	}
	";
} 

---

As you can see it works just fine on most of the different colored balls, but the yellow and orange aren't working for some reason, and the purple is having some weird issues. I just can't seem to figure out what the problem is, but I think I'm getting close.

Generally, you want to avoid if statements at all times in HLSL. Unless you're compiling to 3.0 shader target.

The actual assembly that is output is actually not capable of using the IFs the same way you are using them.

In actually, it always computes what is inside the if, and then uses it based on the conditional.

I would recommend the directX HLSL tutorial at first. You're getting there tho.

(Also, the shaderX2 books are extremely useful for beginners... the introductory one anyway).

Alright, I'll check it out. This really is my first attempt at using HLSL at all, so maybe this is just a little too complex for my first shader.

If you don't know what a dot product is making a toon shader is very difficult. It requires the use of taking the dot product of the light, and clamping the shading inbetween two values...

Personally, it's taken me a long long LONG time to get my understand even remotely aware.

A good place to start (and where you can look at other toon shaders) is:

http://www.coniserver.net/wiki/index.php/Shaders

There is even an HLSL tutorial which is... eh it's ok. There's tons of books on this stuff but your math needs to be good to get a decent understanding.

good luck!

I understand how to calculate a dot product and that it has to do with vectors, but I never really understood why it is important. All of the explenations I've found may just be over my head, but they don't seem to explain anything. Maybe I'm just being a little bit slow, but could someone explain how this is used in a shader? What is the concept? Learning the concept is the most important thing for me to learn things.

Well, in your instance.

You would want to take the dot product between the normal of the surface and the light (the light vector should probably be inversed).

This will give you the angle between the surface point and the incoming light.

Now, what you should do is... map say, 0-15 degree's to the darkest shade, 15-30 degree's to a lighter shade, and 30+ to just regular shade.

Ok maybe those values need to be tweaked; I'm not toally sure.

Now, the big problem is that I don't see any of the other toon shaders in HLSL on the wiki. So what I would do if I were you, is go download ati rendermonkey. It has examples in it. in there is something called NPR metallic or some jazz, (NPR = non-photo realistic). Look at that. It will help; alot. The only way you can really learn is by diving in head first on this stuff until it makes sense.

PS. The shaderX2 books by Engle are excellent and there is an introductory book that covers many facets of directX shaders. I highly recommend it if you are serious about getting into shaders.

hmm, why avoid if-statements for shader model 2? i never recognized huge drawbacks

Well, toon-shaders are easy when you do them for each object. Doing it in a post-processing shader is a bit more of a challenge...

It will take alot more work then if it were in a model material