Maybe some use of vec_to_screen and vec_for_screen is a solution?
Basic idea:
Compute the player position into a screen position.
Set the y values of that screen position to the left and right boundaries of the screen (0, screen_size.x e.g.)
Calculate the world pos from each of thoses border positions and use them for clamping the movement or whatever.
It looks a bit ugly maybe:
...
var border_left[3];
var border_right[3];
...
vec_set(temp,player.x);
vec_to_screen(temp,camera);
vec_set(border_left,vector(0,temp.y,temp.z));
vec_set(border_right,vector(screen_size.x,temp.y,temp.z));
vec_for_screen(border_left,camera);
vec_for_screen(border_right,camera);
// uses the borders for clamping or whatever
player.y = clamp(player.y,border_left.y,border_right.y);
...
edit:
Of course this needs to be computed in a while loop, as the player position may change each frame and thus the borders change.
