It's really just a matter of playing around with the numbers until the ball acts the way you want it to. Reducing friction and torque are a good start but if your level planes have different slopes then obviously the setting that just about keeps the ball from rolling up the steep slope will allow the ball to easily climb a shallow one.

A couple of ideas:
Compare current and previous z position. If z has increased then apply a force from current to old position or even better add a counter-torque that slows down the ball so that it can't move on in this direction. Also check the current rotational speed and stop adding torque if it's above a certain threshold.