1024/1024 is four times as big as 512/512 and therefor the reloading would take also four times longer. Doesnt bother you if you only have to reload one or two textures but if you have 200 of those at a time, it can start to harm your fps very badly.

Imagine every model in your game as a "painting by number" picture. If the gpu has to draw 1000 number fileds instead of only 200, it will take a lot longer for him to finish the single object.
If you have lots of those models it will take a lot longer.

Most people think that faces slow down a game. But thats only partwise true. CPU´s can handle lots and lots of wireframe coordinates at once. They only start to struggle if they have to fill all faces with texture pixels. And the more they add into them the longer it takes to display.

simple example from my test level:
a crowd of people 30 á 2200 faces a character results with a 1024/1024 skin: 66.000faces and a 10-30 frames per second.

same characters with an alternative skin 512/512:
results in 30-45 fps.

Jsut make some test your own to see how much your game would speed up if your simply resize all textures down one number.