I was trying to have massive arrays for a semi-DF clone game, but I always have problems when I go over certain sizes...

for example:

//this works
var localmap[8192][8192][2];

//this just never starts till I get a crash when engine is trying to open
var localmap[16384][16384][2];


If I change variable type I can get bigger sizes but still have problems:

//this works:
char localmap[16384][16384][2];

//this just crashes:
char localmap[32768][32768][2];


I have tried setting nexsus from 30 to 80 and 200, but no option seem to have any effect on what size arrays I can have!

Mabe I am doing it wrong?


I have thought of trying other wierd stuff to be able to load all the data I need, like for example using an image instead and treat every pixel as an x/y of the array and rgb as three char variables for each.
Essentially an image would be the same as an array like this one:
char localmap[x][y][3]
But it would have to be a gigantic image and I imagine there is also a limit to loading that?

But I would prefer to avoid this approach as ideally I would like to have a little more info and RGB+A is much more limited than something like this:

typedef struct TILE
{
char type;
short info;
}TILE;

TILE localmap[localsize][localsize][2];

Anyone's thoughts on this? has anyone ever tried to use arrays this big? How did you get around this?

Just do the math:

4 Bytes (size of var) * 16384 * 16384 * 2 = 2147483648 Bytes = 2 GB

Since Acknex is a 32 Bit programm thats exactly the maximum number of ram it is allowed to use.
(Source: http://msdn.microsoft.com/en-us/library/windows/desktop/aa366778%28v=vs.85%29.aspx#memory_limits)
However, it also needs space for other variables so it runs out of ram (no matter how much physical ram you have in your computer).

I'm afraid an image wouldn't be any better.
If you REALLY need this much data, save it on the hard drive in a fitting format and dynamically load only the few MB you currently display.

Ouch! Thanks for the info. I had no idea about that.

I was trying to avoid having to load dynamically at runtime because with something so big it would take ages to search/read/write from a simple file each time I modified a simple value in it.
The alternative was using a database with sql queries or something similar, so I was trying to find an easy way out.

Having a smaller array in memory solves the size problem but adds a new one... For example, each time I move a row north I would have to delete the bottom row, move all the other rows down a column in the array, and then insert the new row at the top (north).
Even with a small array in memory (lets say 64*64*2) that would mean 128 delete instructions, 8064 move data instructions (8064 read instr. + 8064 write instr.), then 128 variables read from the hard drive, and lastly 128 write instruction to the top row in our small array. And this would be each time the player moves in any direction. This seems like it would create a big speed problem.

I wonder how they solved this when making Dwarf Fortress, One aproach causes memory problems and the other speed problems... Anyone have any ideas or suggestions? Mabe a mix of the two? But how? I am lost...

load areas dynamically, not rows, it might help in your crazy huge world
there is a C++ based game engine where it works fine, using multi-threading somehow, maybe this is why SSE2 required by that.
as I see it's really not an easy task.

You don't even want 2Gb on disk...
Use sparse arrays and store some meta data with your nodes to decrease the size (this might sound counter intuitive, but if you can already describe eg. rectangles in your format, you can store larger areas with repeating content (like water) really efficient).

Oh and do yourself the favor and don't batch everything in one file, thats going to be a pain in the ass. Create a file for every single area in your game and batch eg. 1024x1024 nodes into it

Dividing the map into areas seems to be easiest solution and I will probably take that route. Though I still wonder how games like DwarfFortress or townseed, or others of thet style manage to do it. They can walk between zones like if it was a huge continous world.

A link to that engine you mention would be great, but I am not sure about SSE2. All of today's computers have multithreading, but if I understood SSE2 correctly it is 128bit processing? If so I guess I won't be able to use it, most processors are 64bit now (like mine). Unless by having a dual core 64 bit processor that engine does it's magic on it's own to push 128 bits to both simultaneously... Mabe I have understood SSE2 wrong and am just speaking nonsense. I guess that is probably the case as a 128bit engine seems not very practical as most of the developer's potenciall clients would probably have the standard 64bit processing (and some may even still be using 32bit processors), so I guess I have understood SSE2 wrong?

@JustSid:
That sounds interesting... I am listening. But what do you mean by sparse arrays? What type of meta-data besides the tile type and properties? Could you explain that aporach a little more? Mabe an example? (By example I dont mean code or anything, just a few sentences explaining how that would work)


EDIT:
I think I understood what sparse arrays are, but i didnt quite get how to best implement it. how does this sound to you:
Since not all zones will have predominant 0 value (water or whatever) I thought of first counting the amount of each value, and in the "zone data file" store the predominant (default) value for all tiles that will not be stored. Then the rest of the file would be a list of the rest of the values and the coordinates where they are, separated by some delimiter?

So my file format would be something like:
Dominant_Value|A,coords|B,coords|C,coords etc...

So for a zone like this:
25621
22762
82822
62221
21123

The file logic would be something like this:
2 dominant value
5 in coord 0,1
6 in coords 0,2 1,3 3,0
1 in coords 0,4 3,4 4,1 4,2
7 in coord 1,2
8 in coords 2,0 2,2

and the file itself would look something like this:
2|5,0,1|6,0,2,1,3,3,0|1,0,4,3,4,4,1,4,2|7,1,2|8,2,0,2,2

that would be 55 characters in the file vs the 49 characters needed to represent the same thing in normal format?
2,5,6,2,1,2,2,7,6,2,8,2,8,2,2,6,2,2,2,1,2,1,1,2,3

I think I dont quite understand sparse arrays well, or at least how to store them in a file eficciently?

SSE2 has nothing todo with multithreading or 128bit data types, its actually just a way to do arithmetic really fast by using magic known as SIMD (single instruction, multiple data). This is useful if you have multiple numbers you want to do the same calculation with, for example rows in a matrix when doing matrix multiplication.

And by meta data I mean something like not only storing the actual objects as eg 16 bit numbers in one large row, but instead write some more information together with it. Look at what data you have and search for common patterns and then a way to express this inside your serialized data.
Lets say your world follows this pattern: 01010101010101010101 (where 0 is a tree and 1 is a patch of grass or whatever you want), then you could store the information that there are 20 zeroes and ones alternating each other. Of course this makes only sense for often repeating patterns, and isn't really worth if something like this occurs only a few times.

I just searched for it in the english entry for it on wikipedia and understood SSE2 much better. Check the spanish page wikipedia has for it. It is misleading and makes you think SSE2 is SIMD + 128bit...

it's the Esenthel engine I'm testing beside using and loving 3DGS: http://esenthel.com/?id=news terrain streaming is a great feature of it!
its editors are very cool, ensuring fast and easy workflow, but you really need experience in C++. it's mainly suited for MMO RPG games as I see. so if its features are okay for you, you might do the job with little programming based on example projects. it has a navmesh pathfinder without dynamic avoidance, and there is an example in the tutorials of a grid/tile based pathfinder too. moreover you can extend its grid system due to your own needs (see grid.h in header browser).