Huge array for Deep Learning

Posted By: gtell

Huge array for Deep Learning - 05/23/18 20:28

Dear all,
dear jcl,

I am designing a strategy which uses Deep Learning with a great number of variables and long sliding window.
I have a sliding window of 128 and 163 features.
This means that I would pass to adviseLong an array of 20864 var.
I can't do it, it crashes.
Do you have any other good idea?
I am thinking to pass only the 163 features and then build the sliding window array from R, but the problem is the prediction part.
How to give to prediction the deepness of the sliding window?
Can you please give me some hint?

Thanks.
Best regards.
Posted By: jcl

Re: Huge array for Deep Learning - 05/24/18 10:06

I have not yet seen a machine learning algorithm with 20,000 features, but theoretically there is no hard limit. Make sure that you allocate this array on the heap, not on the stack, and that you use an objective and not the trade result for prediction. If this does not help, you can contact Support with that script and they'll look into it.

For a sliding window, I would use either a LSTM neural net, or indeed store the data on the R side. In the prediction part, use an array for storing the features, and shift it by 163 with any new incoming feature set.
© 2024 lite-C Forums