let me understand, all your models are casting shadows?, if so....that would lower your frame rate, or you are not using stencil shadows at all, but a shadow mapping post proccessing shader....again is slower than just your player having it's shadow on.

Everybody Poops.
here are some tutorials I made.
http://www.acknexturk.com/blender/