R bridge running very slow

Posted By: Dalla

R bridge running very slow - 12/29/17 08:54

I have a weird issue where my R bridge seems to be running very slow. I'm training a DNN using Keras for R with tensorflow backend. Training takes some time, which is expected (three layers with 20 neurons in each).

My issue is that during testing, it takes 5-10 seconds of time to call the neural.load-function.
With R debugging enabled, the console will output something like
"Load KerasDeepLearn_USDJPY_1" and then after 10 seconds or so
"neural.load('....KerasDeepLearn_USDJPY_1')

I'm not that familiar with the bridge, but the long wait between the Load message from Zorro and the subsequent neural.load message from R seems strange to me. I would expect the log from R milliseconds after the call from Zorro.

I've attached my files (.R extensions are apparently not allowed by the file manager, so I renamed it to .txt)

I'm running Zorro 1.74.2 with R 3.4.1

Attached File
KerasDeepLearn.c  (557 downloads)
Attached File
KerasDeepLearn.txt  (485 downloads)
Posted By: jcl

Re: R bridge running very slow - 01/01/18 17:11

We have not observed bad slowness with Keras so far, but you could look into it by adding timer calls to the neural() function.
Posted By: Dalla

Re: R bridge running very slow - 01/01/18 21:47

I added a timer to the neural load function like this
Code:
if(mode == NEURAL_LOAD) {
		printf("nLoad %s",strrchr(Data,'')+1);
		timer();
		int neuralLoadReturnCode = Rx(strf("neural.load('%s')",slash(Data)),3); 		
		printf("nNEURAL_LOAD call time = %.3f ms",timer()); 
		return neuralLoadReturnCode;
 	}


This is the output (only included first five model calls)
Code:
Load KerasDeepLearn_USDJPY_1.ml
neural.load('Y:/Zorro/Data/KerasDeepLearn_USDJPY_1.ml')
NEURAL_LOAD call time = 14981.734 ms
Load KerasDeepLearn_USDJPY_2.ml
neural.load('Y:/Zorro/Data/KerasDeepLearn_USDJPY_2.ml')
NEURAL_LOAD call time = 11214.417 ms
Load KerasDeepLearn_USDJPY_3.ml
neural.load('Y:/Zorro/Data/KerasDeepLearn_USDJPY_3.ml')
NEURAL_LOAD call time = 13523.271 ms
Load KerasDeepLearn_USDJPY_4.ml
neural.load('Y:/Zorro/Data/KerasDeepLearn_USDJPY_4.ml')
NEURAL_LOAD call time = 17167.255 ms
Load KerasDeepLearn_USDJPY_5.ml
neural.load('Y:/Zorro/Data/KerasDeepLearn_USDJPY_5.ml')
NEURAL_LOAD call time = 19805.827 ms


As you can see the calls take 10 to 20 seconds

I should add the the models are small, only ~70kb in size
I also tried setting up a completly new environment on AWS and ran the same script, but the issue remains
Posted By: jcl

Re: R bridge running very slow - 01/02/18 08:56

Hmm, when I run the Deeplearn script with a 2-layer Keras net, I'm also getting about 500 ms loading time. That's indeed a bit slow. I don't know what takes so long.

Maybe you can get some hints about how to speed up loading on Keras or Tensorflow forums - otherwise you have to live with it.
Posted By: Dalla

Re: R bridge running very slow - 01/02/18 10:06

Thanks, good to know it's not an issue on my side at least. I´ve found others having similar issues, but no solution yet.
Posted By: JRA

Re: R bridge running very slow - 06/14/18 10:33

Hi Dalla:

Have you tested the GPU version? I´m running it but at the moment it´s not an improvement over CPU.
© 2024 lite-C Forums