if you only use one light it can be fixed but i think if you use multiple lights it isn't easily possible to really solve this problem since you would have to know which light cast which shadow to handle specularity correctly. (maybe it would work by encoding the light ids in the alpha channel of the light map somehow but i don't know a light mapper which does this. )

i would experiment with thresholds or something like that. it should at least be possible to make it less noticable.

...
the recent realtime shadow mapping examples by jcl have the same problem if you use multiple lights.