Hmm, when I run the Deeplearn script with a 2-layer Keras net, I'm also getting about 500 ms loading time. That's indeed a bit slow. I don't know what takes so long.

Maybe you can get some hints about how to speed up loading on Keras or Tensorflow forums - otherwise you have to live with it.