Two thoughts:

1. Wouldn't model instancing help?

2. Wouldn't it be faster to combine "chunks" you aren't in to one model (build a mesh out of one model) to reduce the draw calls?

MarianFrische.com