One problem I see is that you constantly add to your move number. It should be speed.x = 3 ...
Next is the your character will move like a car. Forward. So when pan and roll are changed so is forward. So you are moving more like a plane or space ship. To solve your pRoblem use richi's idea and some playing around.

Another idea is to record your z position before the movement and then set your mesh back to the z position.