Gamestudio Links
Zorro Links
Newest Posts
Newbie Questions
by fairtrader. 12/06/23 11:29
Zorro Trader GPT
by TipmyPip. 12/04/23 11:34
Square root rule
by Smallz. 12/02/23 09:15
RTest not found error
by TipmyPip. 12/01/23 21:43
neural function for Python to [Train]
by TipmyPip. 12/01/23 14:47
Xor Memory Problem.
by TipmyPip. 11/28/23 14:23
Training with command line parameters
by TipmyPip. 11/26/23 08:42
Combine USD & BTC Pairs In Asset Loop
by TipmyPip. 11/26/23 08:30
AUM Magazine
Latest Screens
A psychological thriller game
SHADOW (2014)
DEAD TASTE
Tactics of World War I
Who's Online Now
7 registered members (3run, miwok, AndrewAMD, Quad, TipmyPip, fairtrader, 1 invisible), 637 guests, and 2 spiders.
Key: Admin, Global Mod, Mod
Newest Members
fairtrader, hus, Vurtis, Harry5, KelvinC
19019 Registered Users
Previous Thread
Next Thread
Print Thread
Rate Thread
Page 1 of 2 1 2
Out of Memory while compiling script defining 3630 element array #432889
11/18/13 16:25
11/18/13 16:25
Joined: Sep 2013
Posts: 504
California
G
GPEngine Offline OP
User
GPEngine  Offline OP
User
G

Joined: Sep 2013
Posts: 504
California
My script requires about 15 constant matrices of floats to perform a few transformations on a handful of input indicators. (If you're curious, the transformation is PCA projection.)

To accomplish this I have tried to initialize the arrays in the code, like this:
Code:
static var pc_1[121][30] = {
  -0.157275083866484, 0.151902882281035, -0.00459335364287939,
    0.0130055542759733, -0.00446267164469449, 0.00249909658366857,
    -0.0466348517770217, 0.0136211757432005, -0.0478944036910219,
...
}



I find I get a compile-time error, Out Of Memory, when the matrices I'm trying to define are 121x30. If I change the program to get by with 15 121x20 matrices instead, the compile step is successful, and the program is correct in operation. So I conclude that the lite-C compiler chokes on initialization of a large array using literals expressed in the code, or that there is a low upper limit on the size of the output binary.

Do you have any strategies for getting around this limitation? I was thinking of reading the values of pc_1 from a text file using file_read at runtime instead. But it will take me some effort to get that right considering the string to float conversion. Before I embark on that, I thought I'd ask, do you think it will work?

My back up plan, which I'm not at all happy with, is to run a background process that is capable of handling this calculation (R perhaps) and have Zorro consult with it (via RPC maybe, somehow) at every bar. Boo.

Last edited by GPEngine; 11/18/13 16:28.
Re: Out of Memory while compiling script defining 3630 element array [Re: GPEngine] #432890
11/18/13 17:08
11/18/13 17:08
Joined: Jul 2000
Posts: 27,967
Frankfurt
jcl Offline

Chief Engineer
jcl  Offline

Chief Engineer

Joined: Jul 2000
Posts: 27,967
Frankfurt
Yes, absolutely, the normal method is allocating the arrays at runtime with malloc() and reading their content from a file. Preferably a binary file. Aside from the memory error, it is not very practical to define 3000 constants in code.

PCA is certainly an interesting approach. Let me know if any coding questions come up and I'll help.

Re: Out of Memory while compiling script defining 3630 element array [Re: jcl] #432891
11/18/13 19:03
11/18/13 19:03
Joined: Dec 2008
Posts: 1,218
Germany
Rackscha Offline
Serious User
Rackscha  Offline
Serious User

Joined: Dec 2008
Posts: 1,218
Germany
@jcl, maybe you could setup a higher limit for the compiler?
In case you want to share those big matrices, it seems to be more comfortable to have them in the sourcefile.


MY Website with news of my projects:
(for example my current
Muliplayer Bomberman,
GenesisPrecompiler for LiteC
and TileMaster, an easy to use Tile editor)
Sparetime-Development

Re: Out of Memory while compiling script defining 3630 element array [Re: Rackscha] #432910
11/19/13 04:22
11/19/13 04:22
Joined: Sep 2013
Posts: 504
California
G
GPEngine Offline OP
User
GPEngine  Offline OP
User
G

Joined: Sep 2013
Posts: 504
California
That explains it.

Do you have an example of reading floats from a binary file?
file_read Reads the content of a file into a string.

Re: Out of Memory while compiling script defining 3630 element array [Re: GPEngine] #432918
11/19/13 11:59
11/19/13 11:59
Joined: Jul 2000
Posts: 27,967
Frankfurt
jcl Offline

Chief Engineer
jcl  Offline

Chief Engineer

Joined: Jul 2000
Posts: 27,967
Frankfurt
int size = 121*30*sizeof(float);
float* pc_1 = (float*) malloc(size+1);
file_read("myvars.bin",pc_1,size);

The allocated pc_1 size must be 1 byte more due to the terminating zero of strings.

Re: Out of Memory while compiling script defining 3630 element array [Re: jcl] #433032
11/21/13 16:56
11/21/13 16:56
Joined: Sep 2013
Posts: 504
California
G
GPEngine Offline OP
User
GPEngine  Offline OP
User
G

Joined: Sep 2013
Posts: 504
California
I have one doubt.
How will the compiler know what pc_1[i][j] means if I'm not telling it the size of the first dimension anywhere?

Re: Out of Memory while compiling script defining 3630 element array [Re: GPEngine] #433049
11/22/13 08:14
11/22/13 08:14
Joined: Jul 2000
Posts: 27,967
Frankfurt
jcl Offline

Chief Engineer
jcl  Offline

Chief Engineer

Joined: Jul 2000
Posts: 27,967
Frankfurt
It won't. A real programmer does not write pc_1[i][j], but pc_1[j+121*i]. Multidimensional arrays are for pussies. wink

But if you want, you can define your array as you originally did, instead of using malloc. It must then however be 122x30 due to the terminating zero of the file_read command.


Re: Out of Memory while compiling script defining 3630 element array [Re: jcl] #433050
11/22/13 08:19
11/22/13 08:19
Joined: May 2013
Posts: 627
Bonn
Sundance Offline
User
Sundance  Offline
User

Joined: May 2013
Posts: 627
Bonn
LOL. Never heard of that. :-)

Re: Out of Memory while compiling script defining 3630 element array [Re: Sundance] #433123
11/24/13 22:34
11/24/13 22:34
Joined: Sep 2013
Posts: 504
California
G
GPEngine Offline OP
User
GPEngine  Offline OP
User
G

Joined: Sep 2013
Posts: 504
California
I am having a difficult time getting the binary input file into the correct format. My floats are read in to Zorro with incorrect values. They have large exponents, like e-315. So my working theory is I have the wrong data size or byte order. But, I've tried exporting from R on Linux as 4 byte and 8 byte floating point values. Zorro on Windows does not interpret them as intended. I am also having trouble in the reverse direction, asking Zorro to print its byte representation of the float. I thought this command "printf("\n%a", x);" would provide me with a nice hexadecimal hint. It prints 0x1.7b1840p+4 for a float value ~ 23.693420 . Sigh. What does that even mean?

Anyway, I'm still working on this aspect of the problem.

Re: Out of Memory while compiling script defining 3630 element array [Re: GPEngine] #433125
11/24/13 22:48
11/24/13 22:48
Joined: Sep 2013
Posts: 504
California
G
GPEngine Offline OP
User
GPEngine  Offline OP
User
G

Joined: Sep 2013
Posts: 504
California
And of course, "file_append("abc.txt", &x);" provides no hints, either. It prints nothing to the file. My working theory there is that the first byte is 0x00.

Lots of guess and check happening here

Page 1 of 2 1 2

Moderated by  Petra 

Gamestudio download | chip programmers | Zorro platform | shop | Data Protection Policy

oP group Germany GmbH | Birkenstr. 25-27 | 63549 Ronneburg / Germany | info (at) opgroup.de

Powered by UBB.threads™ PHP Forum Software 7.7.1