I have narrowed the problem down to the world position VS local position.
But I can't find a way to reset both the world and local position to zero.
(if I set the world position to (0,0,0) then the local position gets changed to the negative vector and vica versa)