@rojart, you better remove HDR and shadows, so only physics will affect on FPS.

I test with my GeForce 6600 GT, so cloth are simulated in software.
I get 10 fps with destroyable cloths and 30 fps with attached model.
But I think, if you'll remove such effects as stencil shadows and HDR, it'll works faster.