In a vertex shader, each vertex is generally only aware of its own position and its normal. There's a bit more than that, but basically you want each quad to rotate around the quad's centre, and in order to do that each vertex needs to know its position relative to its quad's centre, and this cannot be done in a shader.

I would personally do it with bones, if you're using the A8 beta, and do bones animation in the vertex shader. Use ent_bonerotate to orientate the bones when you need to. It'd be worth a try, anyway. It might still be slow with lots of bones, and it'd be tedious to set up. It could be done in A7, but all the vertices will be moved using the CPU, and that'll be slower.

Alternatively, if your Max plugin can put down single vertices instead of quads it wouldn't be much effort to write a function that just places a sprite at each of those vertices -- like Slin said, this'll be relatively slow, but A8 has sprite instancing, which might make up for it.

In short, I don't think what you want to do is feasible in A7, but A8 might have what you need on the off chance that you're a private beta tester.

Jibb