Foxfire,Thank for that,I will be glad to do that.But i dont like msn i use yahoo instead.

we can colaborate (the 3 of us) to figure out a working implementation in GS.Its good idea to do that. I am waiting for you Foxfire and CItrok

I found that my approach may not work when knife(B) is completely inside base(A). in this case wrong results may appear.Some other cases that wrong results appear :
1- when we handle concave-shaped models (as both knife and base)
2- when the camera (eye) is inside either base or knife (which cause the stencil buffer to completely filed)

It has some shadowing and collission detection knot. So for now it is not practical unless we work on it to figure it out.

I think this approach is better:
As always we have to objects(models) on to be subtracted(A) we call this "base" and one to be knife(B) we do following :
1-Render A (base) using a depth map generator shader (simple camera-space Z > pixel brightness) and store result in a buffer for example "bdm" (stand for base depth map)
2-Render B same way and store result to a buffer named "kdm" (stands for knife depth map)
3- Do this :

for each bpx as pixel in bdm //for each pixel in base depth map
for each kpx as pixel in knm //for each pixel in knife depth map
if bpx == kpx //if the depth is equal
fill same pixel in stencil buffer
end
end
end

Now stencil buffer contains area where surfaces of A and B are overlaping

4-Render B with flipped normals using stencil buffer on screen
5-Invert stencil buffer
6-Render A using stencil buffer on screen
7-end

It seems it works better when base is a flat surface and knife is convex-shaped for example it can used for bullet hole ,so on. Any idea?

For collission : Every time C (third object) collides with A(base) and NOT collides whit B(knife) it (C) collides the way we want

For shadownig : I have no idea.

i think i might have a helpful resource. look up Red Faction's Geo-Mod Technology, Red Faction is the game i started my hobby as a Game Artist.

Is there any shader guy who can implement this?

I dont think 3DGS support realtime primitive instancing/creation.
You still can do it using C++ I guess, using directX interface, but it's beyond my knowledge...

There's another option, with Pixel Shader 3.0 Geometry Shader, but I dont know if 3DGS support that too...