Great approach! I like it.

I think if the specularity is less bright in the shadow then it will be ok. There can be some specularity as long as the shadow is not completely dark. And if I understand it right then the specularity will also be less bright in the shadow. So it is fine.