Vertex, Darkinferno:
This is exactly the problem of many indie engines: effective scene management.
The BSP system of A6 was quite nice but took very long to calculate for bigger levels. And it was not able to handle shaders, created too many render batches.
UDK and Source probably have a more modern approach to BSP. They also have to place portals manually. This will help to calculate the BSP tree faster.
A7 got an ABT system that is quite helpful for outdoor scenes.
C4 has a more modern approach. You place your zones and portals manually but then it does not have to calculate any complicated BSP data at all. And it works really fast for indoors with shaders and real-time shadows. You can see real-time lights and real-time shadows without issues through portals.
C4 also has some occlusion portals and occlusion blocks (to block everything behind a certain area).
Unity3 will get this industrial proven Umbra system, it is an occlusion culling system similar to what I already mentioned for C4. It has been used for many big games like Mass Effect as an example. It uses modern hardware, multi-threading and different low level optimizations.
Unity also has some streaming system and it will probably have any data tree like quad or oct tree. I don't know for sure, but the island demo shows, that it works fine even on different systems.
The comment of Darkinferno, that all this can be done in A7 might be right for small scenes. But I am afraid it does not stand if you are looking for bigger scenes. Many of us know from experience that there are problems when you have lots of entities in your scene, there were many stress tests to show this. And there are also problems when you add shaders to bigger worlds. And shaders on BSP blocks are very slow.
And if you want to compare with the upcoming Unity 3 then it will be really hard. The deferred rendering alone will allow you to use way more shaders in a scene and way more lights than you can do with a traditional forward rendering approach.
But I agree, there are some nice small scenes around, made by some skilled shader programmers. I admire this.
