As I re-read your right mouse code, it seems that you want to
move the worldCenter also while keeping the camera at certain
distance, if yes.....

Code:

---

if(mouse_right == 1)		// Pan with right mouse
{
   vec_diff(temp,worldCenter.x,camera.x);
   camera.x -= mouse_force.y * 40 * time_step;
   camera.y += mouse_force.x * 40 * time_step;
				
   worldCenter.x = camera.x + temp.x;
   worldCenter.y = camera.y + temp.y;
   worldCenter.z = camera.z + temp.z;
}

---

If not, just ignore it.