There you render the vegetation (separated from the game world) with alpha testing and use an alpha mask to keep the smooth edges that you normally would lose with alpha testing alone.
Thanks, Felix, but this technique doesn't solve my problem.
I had used a similar technique some years ago, but without any shader. I used two models, which had exactly the same polygons and exactly the same position within the level. One model had a bitmap texture with overlay (alpha testing) and one had a TGA texture with an alpha-mask and soft borders. The result didn't look good, because I still had sorting problems at the edges (the soft borders were sorted in the wrong order). And this doesn't look realistic, especially when I use very detailed textures.
