Hmmm, is there a transparent border on the down or right side the mouse may still be over?
Maybe the mouse is hitting another button before it hits unpopulated panel space.
Maybe the mouse is leaving the panel before/same-time it has left the button.