What I would try is something as follows:
VECTOR cam_angle;
cam_angle.x -= mouse_force.x*4*time_step;
cam_angle.y += mouse_force.x*4*time_step;
cam_angle.z = 0;
vec_set(camera.pan,cam_angle);
ang_rotate(camera.pan,player.pan); // should be the right command, but I oftentimes confuse it
offset camera in vec_rotated negative x-direction