As this is the case, I really can’t use stencil shadows due to the issue they cause when the camera moves into them.
I think that this problem is solved since 6.50 where conitec changed the algorithm of the stencil shadows. But stencil shadows are to slow for things like this, maybe you should look into the upcoming plugin from Chris3D (look into forum "third party tools").
It looks like a good start, but you should really overwork some things. You could use baked shadows for the fixed environment (like the houses) and normalmapping shaders for the walls of the buildings.
Dark_Samurai