Sorry, I had somehow overlooked this bug report. We'll look into that. The mouse mode has most likely nothing to do with it.

-Update: There is nothing broken with vec_for_screen. The difference to A7 is due to a more crude algorithm used by A7, which didn't take into account the angle difference of the position ray to the screen center axis. Therefore you got the wrong -9999 coordinate in A7. This did not matter for coordinates close to the screen, as the inaccuracy is small - therefore no one has noticed this so far. But it matters for coordinates far away, as in your example. In that case, A8 gives more accurate results.

This will be added to the manual.