Also the Shade-C version i'm using atm doesn't really handle alpha channels (transparencies) well.. unless I use hard edges, which is a great downside for the scenario. Therefore my vegetation doesn't look as good as my architectures but instead it flickers in the distance. That's really bad.
That is common problem among all other games out there and is not related to the A8 engine in general.There are several options available when rendering transparent effects with alpha on consumer graphics hardware:
Alpha-testing uses a one bit value output by the pixel shader to determine if the output fragment is visible or not. Alpha-testing integrates well with common z-buffer techniques since all output pixels are fully opaque. However rendering with alpha-testing commonly gives rise to aliasing at the alpha edges, as you already mentioned.
Alpha-blending uses a scalar value output by the pixel shader to blend a rendered fragment with the destination pixel data. This is, when you "just" render your models without anything else... Though, when rendering layers of foliage with alpha-blending, z-buffering artifacts are common. This can largely be resolved if the rendered primitives are sorted to draw furthest from the camera first. Sorting primitives before rendering is usually a prohibitive CPU cost for game rendering and is not supported by Gamestudio, as JCL stated several times throughout the last 10 years here (geez... I'm getting old)
Real-time techniques for order independent transparency are becoming feasible on the latest hardware. Although interesting, these techniques are not yet a realistic approach for current generation games consoles and should als not be considered for PC games as well IMHO!
A third approach is called Alpha-to-coverage, it converts the alpha value output by the pixel shader into a coverage mask. This coverage mask is combined with the standard multisample coverage mask to determine which samples should be updated. Alpha-to-coverage, when combined with alpha testing, gives softer edges without sacrificing the ability to use the z-buffer with unsorted primitives. Although this is an improvement on simple alpha testing, the resulting alpha gradients can be of poor quality compared to those obtained in alpha blending. This is particularly true when using a low number of samples or on hardware that doesn't support flexible coverage masks.
The Alpha-to-coverage approach is used in several games out there and can be used very well with forward renderes like A8 together with postprocessing stages. I do something similar in my SSAO solution for softalpha sprites, but in a different way.
In practice, you render the scene twice or three times, depends on your setup (Images were taken from a whitepaper from the creators of the racing game PURE):
once without foliage:
once with writing out the blended alpha values and with a max operator:
and onces with foliage only (using alpha testing):
Then you combine everything in a postprocessing stage and you get softer edges with (kinda) sorted polygons:
So, theoretically, you could extend Shade-C by yourself or you ask BocHavoc if he implements exactly this technique.
[EDIT]
I have to admit that this works well for large scale foliage. Maybe you have to come up with a slightly different solution for grass foliage, due to the high amount of quads rendered. I could imagine a dynamically sorted model based on bones on the GPU could work out, too.
