Yes, the object is trying to face the given pan as if it is rotated around the world's z axis.
I'm not sure how to get the behaviour you want. Perhaps someone else can try, or I might put more time into it later, but I've got other things I need to get done today.
