I think I got the problem now.

Could it be possible that mouse_dir3d was previously normalised to a length of 1 or at least to a very low value and now it's not any more?

I previously used vec_scale to scale the direction to the length I wanted it to have - assuming that a direction vector is already normalized to the length of 1. When I now use vec_normalize instead, it works fine.