Um... Transparent parts of a button are NOT detected by the mouse. All images are squares, because your screen is a quadrangle (rectangle/square). You can't make a circular image, unless you've got some kind of special software (which I have no clue where to get, or how much it costs). But you don't need that anyway...
Make SURE that the transparent parts are FULLY transparent. If it's 99.99% transparent, the mouse will still detect it. But anything fully, 100% transparent is not detected. I think the best thing would be to use targa (.tga) images, instead of .bmp or .pcx. Targas are of better quality anyway. But it will allow you to have REAL transparent parts, instead of using black for transparent. And you won't need to use "OVERLAY" either, just "flags = SHOW;" (or "VISIBLE" for older versions). I'm assuming you have something like Gimp or Photoshop... God forbid you're using MS-Paint!
But trust me, I've made many a button. You DON'T need a "circular image", or any "alternately shaped images" for your button. A regular .tga with real transparencies is the best thing. Try to avoid using other formats with black RGB(0,0,0) for transparent with the "OVERLAY" flag. It often makes ugly looking buttons with jagged, misshapen edges anyways. I know this info is correct, because I've even used text for a button before. The mouse wouldn't detect the button unless I was directly over a letter, a non-transparent part. So I would make a background layer, and set opacity to 1%, so you couldn't see it, but the mouse would still detect the button between letters.

P.S.- If you're using your own mouse cursor bmap, you might need to set mouse_spot. See manual. You can define which part of the mouse bmap is the "hotspot". Maybe irrelevant for your project, but maybe not...