As far as I understand the manual correctly, the event isn't meant for buttons at all.
Why are you not satisfied with the way how it works?
Do you want that the event only happens when clicking at a certain area of the panel?
Then you can compare the panel's position with the position of the mouse pointer and write a restriction in the event to a certain rectangle of the panel...