I would also think that in fact your WMB is the problem. If I spotted that correctly, there are about 450 map polygons in view when your fps drop - compared with your PC's performance this could well cause your problems.
Try setting all the blocks to "flat" and see if anything changes...