ent_create and memory leaks

Posted By: Carlos3DGS

ent_create and memory leaks - 03/01/12 15:10

Some time ago I learned in thread on this forum that there was a problem with the ent_create() function.

As I understood the conversation, when you ent_create() memory is allocated for that entity, but on removal it is not completely freed, most of it is excluding a small part.

This was a huge problem for me because I dynamically create the world as the player moves along and remove the parts left behind to create a theoretically "infinite" sized world. I thought that this small "memory leack" would be problem in long gameplay sessions since it would gradually stack up and could cause problems with enough time...

Because of this problem I designed a system to compliment the ent_create and ent_remove functions. It is basically a linked list of entity pointers. What it does is whenever an entity is removed, instead of removing it makes it invisible and places it well below the level and adds it to the "avalable entities linked list". Whenever you create a new entity I first check this list, if it is not empty the last entity on the list is re-used using ent_morf, visible flag is set, moves it to the desired position, and removed from the "available entities linked list". Only when this "entity recycle bin" is empty new entities are created in the traditional way.

I was wondering if sprites were also causing this problem (since they are also created useing ent_create)so I wrote a test to confirm if they had to use this system also or not...
Code:
#include <acknex.h>
#include  <default.c>

ENTITY* test[1000];
int i;

void main()
{
	level_load(NULL);
	wait(3);
	vec_fill(camera.x,0);
	camera.z=1000;
	vec_fill(camera.pan,0);
	camera.tilt-= 90;
	
	while(1)
	{
		for(i=0;i<1000;i++)
		{
			test[i]=ent_create("ModernArtSucks.bmp",nullvector,NULL);
		}
		wait(1);
		for(i=0;i<1000;i++)
		{
			ent_remove(test[i]);
		}
		wait(1);
	}
}



To my surprize it seems like it does not cause any memory leack, so I decided to test if it really does cause a small memory leack with models:
Code:
#include <acknex.h>
#include  <default.c>

ENTITY* test[1000];
int i;

void main()
{
	level_load(NULL);
	wait(3);
	vec_fill(camera.x,0);
	camera.z=1000;
	vec_fill(camera.pan,0);
	camera.tilt-= 90;
	
	while(1)
	{
		for(i=0;i<1000;i++)
		{
			test[i]=ent_create("test.mdl",nullvector,NULL);
		}
		wait(1);
		for(i=0;i<1000;i++)
		{
			ent_remove(test[i]);
		}
		wait(1);
	}
}



To my surprize it seems that there is no memory leack. But this was said by a veteran of the forum and I do trust his knowledge and skills. So mabe there is something wrong with my tests? Or I am looking at the wrong variables in the "F11" panel?
The starting values in the F11 panel under "memory MB" column were:
Quote:

nex 9
mem 241
geo 0
shd 0
ent 0
fre 2037

currently this last test has been going for around an hour and these values have not changed and is still running with no changes while I write this post.

Under the column "count of" the values for ent and vis go constantly from 1000 to 0 so it is creating and removing entities.

fps is always around 170 so if you do the math 170x1000
and multiply the time it has been running for x60x60
you get 612000000 entities have been created and removed with no change in the "memory MB" column of the F11 panel during this test.

I wanted to put this out there to get other experienced users' opinions on it.
-Is my test correct to verify an ent_create memory leack?
-Is the F11 panel enough info to draw a conclusion on this or is there a better way to test this?
-Is there any other info in the F11 panel I should be looking at?
-Has the ent_create memory leack been fixed in any patch/version that I am not aware of?
-What are your experiences with this issue?
-What is the pps variable in the F11 panel's first column?
Posted By: Uhrwerk

Re: ent_create and memory leaks - 03/01/12 15:48

Originally Posted By: Carlos3DGS
Is my test correct to verify an ent_create memory leack?

From my point of view yes.
Originally Posted By: Carlos3DGS
Is the F11 panel enough info to draw a conclusion on this or is there a better way to test this?

I don't see a better solution.
Originally Posted By: Carlos3DGS
Is there any other info in the F11 panel I should be looking at?

No
Originally Posted By: Carlos3DGS
Has the ent_create memory leack been fixed in any patch/version that I am not aware of?

Where did you get the information a bug like that even exists??
Originally Posted By: Carlos3DGS
What are your experiences with this issue?

I don't see the issue at all. Your experiment pretty much proves that. ^^
Originally Posted By: Carlos3DGS
What is the pps variable in the F11 panel's first column?

It's the number of triangles rendered per second afaik.
Posted By: Carlos3DGS

Re: ent_create and memory leaks - 03/01/12 16:32

Thankyou very much for all your answers!

Originally Posted By: Uhrwerk
Where did you get the information a bug like that even exists??


I got the info from an old thread I participated in some time ago about these types of "infinite world" ideas:

http://www.opserver.de/ubb7/ubbthreads.php?ubb=showflat&Number=386413&page=3

The interesting stuff on this topic is mostly in pages 3 and 4

For anyone not wanting to go through such a long thread here is what I found more usefull from it about the memory issues:
Quote:
Its very IRRITATING, but not actually ridiculous, as JCL explained to me once.

We'll see how well I remember. Its all got to do with how GS caches its memory.
Im just talking about "level-cache" memory, not video memory...

Lets say you ent_create 'ModelA', and it uses 100K of memory.
Later on, you ent_remove it. This memory is no longer used, but is still cached.
Now ent_create another 'ModelA', and it uses ANOTHER 100K of memory.
Later on, you ent_remove it. This memory is no longer used, but is still cached.

Thats where all the memory goes... BUT, its not really that bad, because if
you do a level_load or a level_free, THEN the memory is completely cleared and released.

But thats no good to ME, because I NEVER DO any level_load or level_free's because
I dont want to lose ALL the game-objects created since the last level_load or level_mark.


Do you want further reasoning about WHY the ent_removed memory stays allocated?
I cant remember sorry, all I can remember is that the memory DOES get used for
other stuff, but never re-used by entities.
I THINK it was something like that GS doesnt know how much level-cache an entity
will need until it is FINISHED loading into cache, and so to avoid trying to load
it into a 'gap' that is too small, and thereby corrupting following data,
they simply avoid the issue by always getting loaded on the end of the cache.

Thats my understanding anyway...


Also I was wondering about if the F11 panel's "memory MB" column was enough info to draw a conclusion on this subject due to this:
Quote:
Im just talking about "level-cache" memory, not video memory...


Anyway... The test has been running for around 2 hours now so over 1,224,000,000 entities have been created and removed withought any changes to the memory MB column. fps are still around 170.
The "memory MB" values remain stable at:
Quote:
nex 9
mem 241
geo 0
shd 0
ent 0
fre 2037


Is "level-cache" memory included in the F11 panle's "memory MB" column?
Posted By: sivan

Re: ent_create and memory leaks - 03/01/12 18:49

due to my little experience, only wmb entities remain in nexus after creating and removing them. I also did tests and had no problems with models and sprites.
Posted By: Uhrwerk

Re: ent_create and memory leaks - 03/01/12 20:53

I'm no expert in these memory managment things. I believe JustSid has a lot of knowledge in this area. Maybe he can add his opinion here?

However, I don't think that memory leak theory is very plausible. If you have a look at the ENTITY struct in atypes.h there is no hint that the amount of memory required is not statically known. Additionally not knowing the exact size of a data structure at compile time is a pretty bad excuse for memory leaks.

The "mem" column should be the amount of virtual memory allocated. You can have a closer look at the meaning of the columns here: http://www.conitec.net/beta/aAnhangC_Tasten.htm

Last but not least:
Originally Posted By: Carlos3DGS
The test has been running for around 2 hours now so over 1,224,000,000 entities have been created and removed withought any changes to the memory MB column.

This is pretty much the prove that even if such an ominous bug exists you're completely safe to ignore it. You won't have such difficult conditions like you created in your test in your game as well, would you? However, keep in mind that your solution with making entities invisible and storing them for later usage will most likely outperform the aproach of always using ent_create and ptr_remove because ent_create is a bit slow. I'd suggest using the ent_create and ptr_remove solution beacuse it is easier to implement, check the performance and if there is any bottleneck switch to your solution with "caching" used entities instead of removing them.
Posted By: EvilSOB

Re: ent_create and memory leaks - 03/02/12 17:06

Well, seeing as this thread is probably all my fault, I had better speak up.

Below you will find a simple standalone script I would like you guys to test for me...
But I need all types of OS and GS versions combinations please... I shall explain...

The following code is RECONSTRUCTED from memory (as best as possible) of the
code that originally spawned this 'memory-leak' idea in my head.
(I know its not a REAL memory leak, but I'll call it that for simplicity).

I originally designed the below code as a CRUDE test of memory consumption
WAAAY back in preparation for a (still un-born) major project.
But, seeing as it failed the test by un-expectedly exposing the memory leak.
This was followed by months of racking the forum, annoying JCL, and much code-hacking.

In the end, there was no known CURE... So I then embedded knowledge of this leak
deep in my brain so I would always program 'around' it, by reflex as it were.

Now its back... as it resurfaced whilst discussing a sub-system with Carlos.

But, then he goes and exposes my belief to be unfounded... The BLASPHEMER!

But ALAS! He was correct! It has taken me TWO days of coding and re-coding to
be able to reproduce my original problem... Then a few more hours to whittle
the code down to something you guys can interpret and test...

WHY was it so hard to reproduce you say?

Well, its been so long since I discovered this problem, that I had forgotten
it was back in A7 days.... I needed to load up and old version of gamestudio(7.80)
in order for the fault to occur... It doesnt appear in A8.

You hear that Carlos? A8 is clear!! You can ignore all I ranted on about...


So here is what Im hoping for from you guys, because I dont have ANY computers
that have NEVER EVER had A8 installed. (I want to exclude possible C-runtime conflicts).

I want to find someone running XP and A7, and someone with Windows7 and A7.
Ensuring those machines have never had A8 (C+ runtime portion) installed...

I suppose some A8 versions wouldnt hurt, especially if any fail...


Thanks muchly for this mission, should you choose to accept it...

And my apologies if my flawed (or at least outdated) beliefs have cause you grief...


And so... the code ...
Just run it and hit F11 to bring up the stats panel. No external files needed.
Once you have observed its pattern of processing and memory consumption,
hold either SHIFT key to enable ent_clone which triggers the leak.

If it is LEAKING, the memory will climb and never come down, even if you release shift,
and it MAY even reach a point where acknex crashes with an 'out of memory' error.

But if it is OK, holding shift will make the memory clime to a peak and hold there,
and the memory will 'slowly' fall back to a nominal value when you release shift...

Have Fun....
Click to reveal..
Code:
#include <acknex.h>
#include  <default.c>

#define BANK_SIZE		1000
ENTITY* test[BANK_SIZE];



void main()
{
	max_entities = 1000;
	level_load(NULL);		wait(3);			vec_fill(camera.x,0);	//fps_max = 60;
	camera.z=100;			vec_fill(camera.pan,0);						camera.tilt-= 90;
	//--------------------------------------------------------------------------------------
	int l, i, count=0;	for(i=0; i<BANK_SIZE; i++)		{	test[i]=NULL;	}
	//
	while(1)
	{	for(l=0; l<20; l++)
		{	for(i=0; i<(BANK_SIZE/20); i++)
			{	
				// create a random model
				
				if     (random(3)>2)	
					test[i+(l*(BANK_SIZE/20))] = ent_create(SPHERE_MDL, nullvector, NULL);
				else if(random(3)>2)	
					test[i+(l*(BANK_SIZE/20))] = ent_create(CUBE_MDL,   nullvector, NULL);
				else
					test[i+(l*(BANK_SIZE/20))] = ent_create(SHADOW_DDS, nullvector, NULL);
				//
				//-----------------------------------------------------------------------------
				//
				// hold shift key to enable ent_clone, this triggers the problem...
				if(key_shift)	{	ent_clone(test[i+(l*(BANK_SIZE/20))]);		}
				//
				//-----------------------------------------------------------------------------
				//
				// systematically ent_remove entities that have been around for a while
				if(test[i+(cycle(l+1,0,20)*(BANK_SIZE/20))])
				{	ent_remove(test[i+(cycle(l+1,0,20)*(BANK_SIZE/20))]);	}
				//
				count++;
			}
			DEBUG_VAR(count, 			100);
			wait(1);
		}
	}
}


Thanks again...

Posted By: Uhrwerk

Re: ent_create and memory leaks - 03/02/12 23:38

Cannot see any problems with the above code - as you foretold. Latest beta version.

Even if the above code would show a problem, Carlos isn't using ent_clone so the problem wouldn't have hit him anyways.
Posted By: Carlos3DGS

Re: ent_create and memory leaks - 03/04/12 08:35

I have A8 installed on both my computers so I don't have one that has never had A8 installed.

Anyway, the test with A8 withought shift key remain stable at:
Quote:
nex 3
mem 241
fre2036

Holding the shift key have a slight variation but as you said goes back down and dosn't build up:
Quote:
nex 3 (stable)
mem 244 (goes constantly to 245 and back down to 244)
fre 2032 (stable)


Thanks alot for going through the trouble of recreating your original tests!
And thanks for clarifying about the a7/a8 difference, I had no idea this issue had been fixed, and am sooo glad I upgraded to a8 a few months ago!
When I saw the old thread I was still using A7 and have always been working around that problem ever since. I never actually use wed in any of my recent concepts, all of them load things by code: models, terrains, heigtmaps, etc... With only one level_load(NULL) at the start, so I had this problem in every single project.
It was just recently (I'm incorporating a sprite based minecraft idea to remove un-needed faces to increase world size even further and improve performance) that I decided to test if the problem also affected sprites, I was confused when it didnt so I re-tested models (thus this thread, reviving an old topic).

Thankyou EvilSOB, Uhrwerk, and thankyou A8! This makes my life sooo much easier from now on! And knowing ent_clone isn't a problem either is great news too! One less thing to worry about in my crazy projects!

Bye bye my dear "entity recycler", you were a pain in the a$$ to work with and will NOT be missed! R.I.P.
Posted By: Spirit

Re: ent_create and memory leaks - 03/04/12 14:21

BTW, Im pretty sure there was no ent_create memory leak in A7 either. I had A7 and have never seen such a leak and if it existed, it would have been reported to Conitec and appeared on the bug list.
Posted By: Uhrwerk

Re: ent_create and memory leaks - 03/04/12 14:59

EvilSOB is an experienced member of the community and he usually knows what he's talking about. If he reports something it is almost certain that there really was a problem. However, I think the original bug seems to be connected to ent_clone more than to ent_create. Maybe there was an issue with fragmentation of nexus memory? I'm guessing here, but I think what EvilSOB mentioned here was not a bug but more a "feature" in the form of a rather unfitting memory managment algorithm.
Posted By: EvilSOB

Re: ent_create and memory leaks - 03/05/12 08:10

Thanks for the compliment Uhrwerk, and your 'description' of the issue is far
more accurate and descriptive than any Ive given to date.

I know from previous / lead-up discussions with Carlos that he was
DEFINATELY going to be using ent_clone in the FUTURE, thats why I warned
him of this issue. Yes it IS ent_clone that has the problem.
It is quite possible I omitted telling Carlos it was ent_clone specific,
and left it under the GENERALIZED heading of ent_create...
(Sorry Carlos, if I gave you a false impression there.)


When I first discovered this issue, it took a long discussion / argument with
JCL before he convinced me that is was not a BUG, but an ARCHITECTURAL LIMITATION.
Therefore no bug report... And I guess thats also why there was never any
mention of it being 'fixed' by A8, cause it was never officially 'broken'.
Believe me, I TRIED to get it on the bug-list, but JCL convinced me otherwise.


Spirit: If you dont believe there was ever this issue, load up an old A7 and try my code.
(yes this will work 'after' A8 is installed, but you may need to re-install
A8 after doing the test.)
If you dont believe me or my words, maybe you'll believe your eyes... laugh


One personal drawback from this issue being fixed is that I can now throw
away about 6 months of difficult programming development, dammit...
But, in truth, good riddance to it. Back to the drawing board...

And thank you Carlos3DGS, for noticing how obsolete I have gotten...
Time to drag myself (kicking and screaming) into the 'A8'th-century.
Posted By: jcl

Re: ent_create and memory leaks - 03/05/12 09:10

I've seen this discussion, maybe I can shed some light on that issue.

A memory leak is a bug and no "architectural limitation". We have no records that anyone found a memory leak with ent_create, and I don't think that such a leak existed. The ent_create function is more or less unchanged since A4.

There was however indeed a memory leak reported with ent_clone in A7 version 7.73 - this seems to be the leak that you meant. I can not remember a long discussion, but it was a clear and obvious bug that was fixed in the next version, 7.75. You can find it in the bug list in your A7 manual. We're grateful for notifying us of such bugs.

So if you hear rumors of some bug on the forum, better communicate with us rather than spending months of heavy programming for working around it. We might save you some time.
Posted By: EvilSOB

Re: ent_create and memory leaks - 03/05/12 19:58

JCL:

The reason this 'bug' has re-surfaced in the community topics is entirely MY fault.
When I first found it, back in 2009, the discussion I had with you (in the latter
part of THIS thread) left me with the ASSUMPTION it was not a bug, but a known limitation.
(as you can see from my final comment in my final post there)
I see now it wasnt a LONG discussion, it just seemed it at the time.
And it was a long time ago, memory can play tricks on you...

So, because I thought it was not a bug, I never looked for it being fixed.

So when I saw someone heading towards a (percieved) pothole,
I naturally warned them of it.
Unfortunately, I was not aware you had fixed that pothole...

Again, apologies to all...

Im just going to have to be more consistanrt in my checking
of the updates in the reported-bug-list.
Posted By: Carlos3DGS

Re: ent_create and memory leaks - 03/28/12 13:49

It's been a while since I had time to continue with my work and browse the forums.
I just came back and wanted to thank everyone that helped clarify this, and also thanks to jcl for personally adressing this topic.
© 2024 lite-C Forums