...Let me look into my shader for more details...

First option would be to reduce the alpha of your normal map. The specular channel does not control specularity (strange, I know). Instead, the alpha channel of the normal map is used to give you per-pixel access...