Why you don`t set only the pan from the camera?

camera.pan = player.pan;