FOV = Field of View

Basically you'd setup an animation for, say, every 30 degrees. With a 90 degree FOV, you'd have an "imaginary" grid of views...



You would create a vertex animation for every viewing direcion on the grid (49 frames). Then you would do frame blending depending on which direction you were facing along the grid. It will take some time to perfect, but it will work...