The amount of force on an object can not be limited, but you should set its speed limits to avoid explosions. If possible don't apply one huge force for the duration of a single frame- instead apply smaller forces of the duration of a few frames.

When objects suddenly jump off it's usually due to collision detection problems (edge on edge) or by forcing the object in unnatural positions (such as ent_creating a new model inside a physics object). In case of the flippers I think the problem is that even after an impact the flippers still keep rotating. The ball has not had a chance to get moved out of the way yet the flippers keep rotating towards it and end up inside the ball. Try halting the rotation for a few frames after hitting the ball.