mouse_dir3d*((camera.z-plane.z)/mouse_dir3d.z)+camera.xyz = position

Lets assume that you levels origin is within the plane and that
the camera is placed at (0|0|100):
                                    0
mouse_dir3d*(100/mouse_dir3d.z) +   0 = position
                                  100

                        1
Now, if mouse_dir3d is  1 / sqrt(3), the whole thing with values 
                       -1
looks like this:


 1                             0   -100
 1 / sqrt(3) * -173.205081 +   0 = -100
-1                           100    200

As you see this is wrong, as the correct position would be
(100|100|0)... Inverting mouse_dir3d would give correct results.
I wonder where I missed the stupid sign -.-


This however, should then just work:

void traceMouse(VECTOR *pos)
{
  vec_set(pos, mouse_dir3d);
  vec_normalize(pos, -1.0);
  vec_scale(pos, camera.z/pos.z);
  vec_add(pos, camera.x);
}