This is due to low friction and low damping settings. Increase those and the object will resist movement.