You should set mouse3D.z to some high value (like 10000) before converting the vector from screen coordinates to world coordinates using vec_for_screen().
Because the .z value represents the "depth" of the position relativ to the camera.
This means: the higher the .z value the farer is the position away from the camera (in the direction the camera is looking)
As you are using lite-c you could also make use of returning a value as you are checking if there is an entity at the mouse position.
int traceToMouse()
{
VECTOR* mouse3D;
mouse3D.x = mouse_pos.x
mouse3D.y = mouse_pos.y;
vec_for_screen(mouse3D,camera);
mouse3D.z = player.z;
c_trace(player.x,mouse3D.x,IGNORE_ME | IGNORE_PASSABLE);
if(you != NULL)
{
return(1);
}
else
{
return(0);
}
}
// example of using the above function:
function check_MousePos()
{
int a = 0;
while(1)
{
a = traceToMouse();
if(a)
{ beep(); }
wait(-0.5);
}
}
// it checks every 30 seconds if there is an entity at the mouse position