Because they don't calculate from the window position but from 0,0 real screen position. At least this is what my tests show. mouse_dir3d, mouse_pos3d, they all depend on mouse_pos which is not added to the window_pos. Pardon me if I am mistaken.

Isn't this the difference between mouse_pos and mouse_cursor? One is absolute coordinates (screen) and the other being relative coordinates (engine window)?

After 4 years of working with the engine I still get confused on what the actual difference between them is?

If that is so... how do I replace mouse_dir3d then?

All mouse coordinates are always relative to the upper left corner of the client area of the engine window, ranging from 0 to screen_size.x/y. This is no "inaccuracy".

http://manual.3dgamestudio.net/ampointer.htm

Found my problem; was not using USE_POLYGON in the trace, resulted in strange 3D positions with even more strange conclusions.