Well, you could still do this with vertex animation, but it would be very difficult. Basically you'd need to think of your FOV as a matrix, or a grid.

Depending on where the desired view angle is, you would have to blend your animations to the target frame on that grid.

...If needed, I can (try to) provide illustrations.