crash after level_load

Hey guys,

after having sporadic (very, very seldom) crashes after a level_load, I was able to get a savegame where I can reproduce the crash every time.

It's always the same level I load that crashes. I haven't had a crash with another level.

The strange thing is that this level only crashes when I load it from a certain other level. It's always those two and in this order.

If I load the level that crashes from any other level I don't have a crash at all.

The crash appears with the first wait(1) after the level_load.

Also strange is that the crash disappears once I remove a random entity in WED in the level where the crash appears ... it doesn't matter which entity I delete.

All levels are very, very small (every room in the mansion is a level).

Unfortunately I cannot use the SED debugger as its internal XML structure for the variables gets too big for a 32bit application and leads into a SED crash. I asked JCL about that a while ago, but it doesn't seem that we get a new debugger :-(

Does anybody had such an error before? I'm really getting mad after trying to solve this issue for a long time.

Any hint is appreciated.

Regards,
Pegamode.

Hi!

I'm afraid that it could be a memory leak (if it doesn't crash when you remove a random entity etc).
Hopefully I'm wrong, cause it might be really hard trying to find it.. :<


My best wished and regards!

Hi,

yes, that's what I think, too ... but I wonder why it just appears with those two levels.

We have more than 40 rooms / levels, but I never noticed a crash like that with other levels. I would expect that I get more crashes like this if it is a memory leak?!?

Any hints how to search for such a memory leak? Or is there a way to use another debugger with lite-c? As I mentioned I cannot use the SED debugger.

Maybe you have used something unique (not used in other 38 rooms?) in those two levels, which might help you to find a bug (just a wild guess).
I really wish I could help with an advice, but I never used any debugging tools. Let's wait till someone more experienced will give us an advice.


Best regards.

I tried to break down the code by using "engine_gettaskinfo" to list all running functions before the level_load and after the level_load (right before the wait(1)), but it didn't help at all. None of the running functions seem to lead into that problem.

If you get this crash from just one level that's mean the crash appear from your action code . To be sure just remove the actions from that level and test ,if the crash gone then you have to locate the problem from your script.

pegamode@ I recommend you to make a little research on the forum (I'm pretty sure you already did, but you might missed something). F.e. Alibaba faced some troubles with crash as I remember (check topics that he created), f.e. take a look at this topic:
No Error just a crash?

I hope this might help. Best regards!

@Dico:

I already removed all actions, but that didn't help. Also all actions write log messages at start, but there's nothing in the logfile. The engine crashes right after the level_load once it reaches the wait(1). If the bug were in the action scripts I guess the crash would appear every time and not just when the level is loaded from a specific one.

@3run:

Yes, I searched a lot on the forum, but none of the things I found helped. My first thought was a broken sprite or model, but I didn't find one yet.

But I'll do some more research on the forum.

I remember this.. uhh..
Did you end all functions which were running in created entities?
I used the PROC_GLOBAL state sometimes and then this shit happened.

At least I hope so :-)

I use PROC_GLOBAL a lot, but I also use variables like levelIsLoaded and levelIsReady to properly start and stop entity functions.

I also checked by using "engine_gettaskinfo" to list all running functions and there's none in the list that shouldn't run.
Unfortunately a started function is only listed by engine_gettaskinfo after the next wait(1), so the list might not be complete.

The thing is that I don't know if it is a function that was started right after the level_load that lead into the crash or if it is something in the level. And I wonder why it just happens in this combination of those two levels.

You didn't specifically state what kind of crash do you get, did you?
acknex has stopped working, script crash in ..., ? Have you used sys_marker already?

It could just be an action (a function) from a previous level that continues to run after the level_load, which then deals with an invalid pointer (not necessarily an entity). The error would be to look for in the level from which you switch to the seemingly crashing level.
I was recently experiencing similar problems with entity functions not being stopped on level_change. Do you have let's say a rocket that shoots high into the sky or some random bird flying high? If they leave the level(_ent)/ ABT boundaries this could lead to problems later on.

I just get a "acknex stopped working" windows error.

Nothing logged out.

If it is a running script shouldn't I see it in the list of running functions? And why does it not crash when I load another level then? And the same question vice versa. If it is a problem with a function of the new level, why does it not crash when it gets loaded from another level ... really strange.

Why does it not crash anymore when I delete a random entity in the level?

I don't know, sorry. Just trying to poke you a little to get you on the right track.

I've once had "acknex has stopped working" errors after level_load in a multiplayer skateboard game. The reason was a faulty texture in a very old WAD file.

How did you track that down? Just try and error?

My first thought was just to modify the level that the crash disappears, but I fear that it would just lead into another sporadic crash in another situation as it doesn't fix the issue itself.

Is there any chance to get an external debugger to work? Unfortunately the SED debugger can't help.

I would advice you to create a blank new level, instead of the one that causes the crash after loading (just save this one somewhere, so it doesn't get changed/damaged). Then start putting everything from the older map into it one by one (yeah, it will take hell of the time, but probably will help you to track the crash down if it's related to something on that level).

Best regards!

Hm... I think I - after jcl told me that the reason very likely could be a faulty texture - just took the level, deleted maybe half of it and tried again. No error -> deleted less and tried again, until I had it isolated (it was the r25.wad if I remember correctly, should anyone else be using that).

When you have let's say such a faulty texture loading that may not cause a crash, but some other object that gets created later could write into invalid memory areas because of that - at least to my knowledge.

had such an issue once , turned out it was something todo with regions (i cannot remember specifically ) , if you use them , try deleting them .

We don't use regions.

Is there any way to identify a faulty texture properly (I don't mean just by replacing them one by one, but something like open it in Photoshop and get an error message)?

Keep in mind that the order of the functions matter for pointers and other calls, so deleting a random entity changes the order of the running functions, so it could be related to a pointer. I'm not saying it IS, only that it COULD BE
Also, another level with the same functions could be starting them in a different order, and again, bypassing this pointer problem...

The question is: How can I track this down?

Unfortunately the engine gives no hint on what it is currently processing. Until now I was able to add several log messages into suspicious scripts to identify the bugs and fix them. Now it seems the problem could be everywhere: in the code, in the models, textures ...

Well, my first thought would be to start eliminating the functions one by one until the crash disappears.

I mean, if you have different enemies, door functions and the player function, start adding "return;" in them right when they start, so you can see what function causes a problem, if it is a function...
If it still runs, remove the return and add it in another function...

Also, its a good idea to always check if a pointer points to something, even if you're sure it does, before using it... I recently had huge problems with BMAPs because of this even with checks that a pointer points to something...

Actually we stop as many functions as possible before each level_load.

So there are less than 10 functions running in background all the time.
I checked all of them by removing them one by one with no change at all.

Then I removed all actions from the entities of the level that crashes ... no change either.

Question 1): If the issue is "inside" the level (e.g. bad texture) would the engine crash while processing the level_load command? Currently the crash occurs just with the first wait after the level_load.

Question 2): Is it possible to get an ordered functions list (queue) from the engine. If so I could check all functions in the queue. Unfortunately "engine_gettaskinfo" doesn't help as functions are only listed there after their first wait.

The Level isn't emideatly drawn. So the error occures when the level is first displayed. So my guess would be that it is indeed a bad model or texture.
As I remember correctly I had an error like this, too.
I loaded a bad texture via BMAP map =""; and only when I set this bmap to an enitity it crashed.

Yes, crashes often occur when you try to draw something that cannot be drawn I've had issues with BMAPS, but I never tried using bad images in levels.
Make sure all your textures are of the supported formats, a power of two and your video device supports the resolution of the images.

Other than that, if youre sure the functions are not the problem, your only option is to start decomposing the level piece by piece to see where it stops crashing...

Make a backup of the original and start removing models/ textures until it stops crashing.

An easy test is to assign the default texture to every block in your level... Shouldn't take long...

If you used any media_play or things like this for example a Video playing or a Music file. Stop ALL handles before Level_load instruction and retry.

After i stripped down my half game i found out, a tv model with a rendered Video caused the crashes ( in nearly all different functions running ).

Greets

We don't use any blocks in our levels, only models.

And we don't use media_play ... we switched over to fmod.

The problem is that it doesn't matter what model I delete in the level. Once I delete a random model the crash disappears.
But I don't think that this solves the problem. I guess the crash will appear somewhere else. So I'm glad that I have a savegame where I can reproduce the crash everytime ... but I haven't found a way how to find the problem yet.

what you mean by : "I have a savegame where I can reproduce the crash everytime".

The levels (rooms) that lead into the crash can't be reached right from the start of the game. Also I had to find out that it only occurs when both rooms are involved.

I currently have a savegame where one playable character stands in one of those rooms and another character stands in the other room. Once I switch the players the crash occurs after the level_load with the first wait(1).

I could try to place the characters directly into those rooms at the start of the game, but I don't know if it helps. If there's some kind of memory leak / overflow it might not happen then.

did you check the level sizes (bounds) .

also what kind of data is associated / stored for each entity , eg. skills or extra pointers?

if you take away the load game and simply start your game manually at the two levels involved in sequence, does it happen still ?

if you replace the entities with placeholder models without textures , does it happen still?

does each of theses levels have their own scripts , and if so , can you show it ?

do you use the physics engine ?

what is shared between these levels ?

try to start at the entities creation and destruction functions/code , what are you doing there , line for line can you post it ?

The levels are very small (just rooms in a mansion).

I tried something like that (mini main function):

level_load([first_room]);
wait(-5);
level_load([second_room]);
wait(-5);

In this case the crash didn't occur, but I'm not sure if this is a useful information. If it is an issue with a texture or model inside one of the levels, would this always lead into a crash? I guess that it might also depend on the total memory consumption, doesn't it?

I'm pretty sure that you are trying to access something that wasn't just created yet, and that causes the crash. That can explain why in situation above everything works. You wait for 5 seconds, so it makes sure that everything get's created on the level. What do you have under that -5 seconds? Maybe that's the key to your crash?


Greets

I also think that it's either a memory overflow or an invalid pointer.

My problem is that I've done the following things without success:

- listed all currently running functions (before and after the level_load) and checked them
- removed all actions from the models in the level where the error occurs

But I still believe that it's like you said ... unfortunately it seems not possible to list running functions before their first wait command. So their might run a function that I don't see at the time of the crash.

I would recommend you to 'disassemble' your level (don't forget to make a backup). Use 'default' texture instead of all textures that your second room has (if your levels are small, this won't be a problem I guess), use 'CUBE_MDL' for models that you create on fly via script, and use simple cube model (just create it in MED, simple cube) instead of all models that your second level has. If it won't crash, then start adding each model one by one. Still no crash? Same for textures. This will help to find out, if it's related to script of models/textures. If at the end, you will end up with getting your level back as it was originaly, then it's 100 % related to code. I'm pretty sure that there won't be easy way out of this.

Greets

I tried to delete the models in the level one by one, but it doesn't matter which model I delete the error doesn't occur anymore.

I even changed the order of the models inside the level and the crash was gone.

So I'm not sure if replacing the models by a dummy will help at all.

Are they the same? Or all those models are different?
So maybe this causes the crash? Imagine having two entities (just a simple example I have in my head) one created after another. The first one tries to acces the second one, but it's not created yet. So you changed the order, and it worked? I'm pretty sure that your problem that causes a crash is complex than what I just said, but I wanted to explain how I see it as simple as possible. Just a thought.

Are there any models sharing same action/functions?

Greets

All models are different.

Usually the rooms consist of:

- room model (walls, floor, ceiling)
- dummy light models (an invisible simple cube model)
- dummy character position models (an invisible cube positioned at doors)
- dummy cam models (an invisible cube model we use to place, move and switch the camera)
- environment models (doors, windows, flowers etc.)

I removed all actions from the models in the room that crashes, so there shouldn't be a dependency that can lead into the crash (at least I hope so).

Yes, a lot of models share the same action.

All models the player can interact with have an action called "useable_object". But this one was also removed in my tests.

So you just have a level (testing one) which is fully made of models and without any actions assigned (just some models placed in WED). And after removing all actions from those models/sprites it still crashes, but when you change order of those models the crash is gone? If you wait for 5 seconds before and after loading level, crash is gone too, am I right? Do you have anykind of loop/looops where you controll entities? F.e. I had script where all door where controlled in one loop, do you have anything like that? Maybe that 'loop' is trying to access a pointer (pointers) that doesn't exist yet, so that's why 5 seconds delay helped. Changing the order could help, so you move model (models) that caused the crash up in the WED list. Anyway, if you have a crash on loading a level made of models without any actions assighed, this is strange.. A

Edit: it's really hard trying to 'guess' what causes the crash, without knowing how your setup of the game looks like.

yes.



no.

The 5-seconds helped only when I load the levels "outside" of the game. I used a simple main function that does nothing but loading the levels one after the other. Even a wait(1) is ok in this case.

I use this function to print all running tasks:



I receive this result right before the level_load:



And this is what I get right after the level_load (before the wait command):



The two entries with "funcname was NULL" are the "player_movement" functions from above that are still in the list until the next wait(1). At first I thought that those cause the crash, but I also removed them completely already ... no change at all.

I also checked all those function in the list, but couldn't find anything suspicious.

As far as I know functions are only listed by engine_gettaskinfo after their first wait command, is this right? In this case the list might not be complete.

If using 'wait(1)' in a small main function that had nothing to do with your project and crash was gone, why you mentioned 5 seconds, it really disappointed me. If you loaded those models/levels aside from your project, and they didn't cause any crashes, isn't it obvious that crash caused by your script (this is at least what I think)? It will be hard to help fixing your crash without knowing how your code looks like and how you setup everything.. Maybe you could try to reproduce it in a small example to show us? Maybe you will even end up fixing it at the end. As I already said, I'm pretty sure there won't be an easy way out of this. Relying on debugging tools won't help in this situation, as you already found out by yourself. But anyway, trying to guess what might cause a crash in a complex project like yours is like trying to drive a car with a bandage on your eyes (or even more harder ).


Edit:Here:

Greets

Sorry for the "wait(-5)" ... it was just something I also tried inside the complete game code to asure that everything is created. Actually I set proc_late for the load_new_room function and use wait(1) or wait(2).

I also believe that the error is somewhere inside the code, but I'm just confused that changing the order of the models inside WED removes the crash.

I'm just hoping to get some fresh ideas where to look by brainstorming with you (by the way ... thanks to all of you for discussing this issue with me).

Maybe I should also check the entities in the "engine_gettaskinfo"-list ... maybe one of the functions get an invalid entity pointer? The question is why should that happen just in the case the rooms are loaded in this order.

By the way ... I'd love to share more of the code with you, but it's really a lot of code. Far too much to post it. And the whole project is about 1 GB.

Maybe the crash just from save game !!!

I wish I could help you more, but I'm out of ideas right now. I'll try to think more of this, maybe I'll come up with something. I hope things will get solved soon.

Edit: it might have to do something with 'proc_mode' maybe, just a guess. Maybe some of those functions which aren't terminated on level change using it?

Best regards.

I searched our bug tracking system and found a bug report some years ago from one of our beta testers.

He reported an engine crash and the attached log file looks quite the same as the one I currently get.

The only difference: The first room was another one (but the one that was loaded and where the crash occured was the same).

Good: So I guess I don't need to search in the first level.

Bad: If it's in the game code, why is it always the same level where this sporadic crash occurs ???

Hmmm ... strange.

I think my next step is to place both characters in the rooms where the game crashes in my savegame and check if I could reproduce the crash right after starting the game. Usually who have to play the game about 45 - 60 minutes to reach those rooms. So there happened already a lot of stuff at the point of my savegame.

Can you load this room without game_save and , how you save your level ?
Can you test by load your level with just game_save and without level_load function ?

you can view memory usage in windows tadk manager-memory ..

have you checked your nexus reserve ?

thats all i can think of given what everyone has thought of already .

good luck

I don't use "game_save". I've written an own function that fills a struct with all needed data and then write it into a file (and vice versa when loading the savegame).

I finally found the time to place the characters into the rooms that lead into the crash right at the start of the game.

Result: the crash still occurs

So at least I can say that it's no issue with the savegame, neither it's something in the characters' inventories nor something that happened during playtime.

I hope to find some more time on the weekend to dive deeper into the code.

I wish you to find and fix everything that causes this crash as soon as possible! Unfortunately there is nothing I could help with. Keep it up man!

Best regards!

The bad thing is that I can even add a freely chosen model to the room and the crash is gone ... so how do I know when I have really fixed the issue ???

Have you just tried resaving all models and esp. textures involved in those rooms?
Or maybe it's something else, like a globally (pre-)defined BMAP or a globally (pre-)defined sound.
Does the level crash if you just load it in a different project? Probably not though, even with a faulty file.

Unfortunately the level only crashes if I load it after this other certain room. In any other combination the level doesn't crash.

Once I remove or add a model in this level the crash is gone.

Does "resaving all models" mean just open them in MED and save again?

Yeah that's what I would try. Although I still find it weird (even though I've experienced it myself before) that files can be damaged.

Some other idea, are you using PNG textures or *.jpg even?

Resaved all model in this room, but no change at all.

I don't know what kind of textures our modeller used ... the room was build with models only.

So the textures are saved in the model and not externally? Maybe you can check the format in the Skin Settings.
Either way, you're a little closer to isolating the problem now.

After resaving all models and textures in game folder, one of the things i would try is to comment each line of code where you play sounds. I only once had this problem in my game (engine crash without any error message and without any warning) and i found after pain search that i simply put one line for playing sound on wrong place which causes crash. The only luck was that this was happening always and only in the same level, and it helped me a lot. Also with this you will check did you maybe have damaged sound file. In your case i don't think that problem is in damaged file, it should probably show engine error message but with this kind of crash you simply must eliminate this possibility.
Whether there is any previous indication that crash will happen, such as a sudden jump in consumption of memory in debug panel or something?

Yes, the textures are saved in the model. In the skin settings I just see the texture size, but not the format.

We use FMOD to play all sounds and music ... at least the fmod debugger showed no problem. If there was a way to debug the level and code it would be much easier. I didn't really get it why the SED debugger uses an XML data structure internally :-(

Then add way more diag() and diag_var() commands to your code!

Almost every function in the project writes log information when starting, exiting and several information while running.

In case of this crash the engine just crashes once the first wait(1) after the level_load is reached.

I'm already logging out all currently running functions between level_load and wait, something like:



But that didn't help. I removed all of those functiona, but the crash still occured.

The only way to get rid of this crash was to add or remove a model in the level, but I'm sure that the crash will appear at another point of the game, because I did quite the same some months ago. I had exactly the same crash, just the starting level was another one, but the level that crashes was the same. Then I changed the order of the models in WED and the crash didn't occur anymore ... until now.

so the crash from your script , for exemple when the model 1 load his action and in his action there is an entity pointer not loaded in level so it will give a crash but when you change the model order it work , try to add someting like : while(!entity)wait(1); in all model actions.

I already removed all actions from any model ... no change at all.

i suggest you upload this level to someone here you trust , if theres no scripts, and basicly its the level that has the problem from what i gathered you have been saying , then that is the only way i see you getting real help .

Currently I'm trying to remove as much of the game code as possible to track the issue down.

I just noticed that at least WED doesn't seem to like long level names.

#define PRAGMA_LEVEL "upper_library_floor.wmp";

leads into an "invalid name" exception when loading a level in WED.
Does someone know if the engine has also a limit here? I just want to keep sure that the crash doesn't come from those long level names.

well if it is limited to the old 8 characters filenames then something like 000_000.wmp to 256_256.wmp would give you 256*256 possible mappings which can be case evaluated to internal more suitable string names , just thinking out loud...

At least 16 characters plus file extension is no problem. The question is: Is it a limitation in WED only or is also the engine affected?

using google advanced search with the online manual as site , i found a mention of it in an old a7 version :

PRAGMA_BIND and PRAGMA_LEVEL now support file names with spaces; still, it is absolutely not
recommended to have file names with spaces or foreign characters in your game.

do not know if it changed ..

oh wait ...here we go

Gamestudio and lite-C support numerous file formats for importing and exporting artwork and data. !! Keep
file names short - the names of entity and texture files (including extension) in WMP and WMB levels are
limited to 30 characters , and texture and skin names in WAD and MDL models files are limited to 15
characters . File names must not contain any special characters except " _ " or " +". Never, ever, use spaces
in your file or path names! Many import filters from other editors will not find texture files when they have
spaces or exotic characters in their name.

http://manual.3dgamestudio.net/fileformats.htm

I neither have spaces nor special chars in my filenames.

Once I use more than 16 chars plus file extension I get an error in WED when I use PRAGMA_LEVEL ... the engine seems to have no problem with it, but I'm not sure if there might be some hidden memory issue.

i found it , previous reply..

on the subject of character limits , also actions are limited :

Actions are another special kind of functions. They don't have parameters and don't return a value, but will appear in the action pop-up list in
WED's entity property panel. So actions can be assigned to entities through WED, and are automatically executed after level loading. Action
names are limited to 20 characters. Besides that, there is no difference between functions and actions.

just in case.

okay maybe this will help out , check out "last_error" in the manual :



also i found that most "acknex has stopped working" reports on the forum , was related to textures/models files , so a test could be to see if every one opens up fine in med-engine preview , and also a ent_create test for each of those used in the level might reveal any possible ones that may cause a problem (if any at all), plus , removing all of them from the level and adding them back individualy and testing each with level_load might reveal any that could cause a problem ..

I tried "last_error" already, but the engine crashes once it reaches the wait(1), so the last_error block is never reached.

Removing entities from the level one by one doesn't help as the crsah disappears once I remove or add an entity from the level. But I'm sure this is no fix at all as the error will just appear at another point of the game (if it is some memory issue).

mmm that sounds like completely bizzare behaviour, i thought it might have been a shared material or texture or script between them ,but adding a random entity to it takes away your crash , makes no sense to me , except what sounds logical is a memmory related problem as you mention , or an engine bug , have you tried out asking the dev's .

i mean what would adding/removing a random entity effect in the level itself before loading , all i can think about internally ,during loading ,would be the entity list , memmory ,pointers , shedule list , materials , textures , bounds ,hulls , stack sizes perhaps , the list could go on but only the devs would know what is internally effected.

I just asked JCL for help on this ... maybe I can get some more information about what really happens inside the engine with the first wait(1) after a level_load.

Lol, 8 pages now

Are you sure this isn't just a one time compilation problem? I mean, once you add or remove an ent, you compile again...and the crash disappears.
Try recompiling without touching it, and see if the crash still occurs.

Only recompiling doesn't help.

But changing the order of the entities inside WED helps. If I just delete a model, click undo and then recompile, the crash is gone. By deleting and undo the deleted model is set to the end of the list in WED and that helps. Of course deleting the last model in the list doesn't help as the order wouldn't change.

I can't fully remember, but I think there was a limit to 22 characters for action names or the like.
The mean part is: actually it's 21 characters (guess the \0 string end marker counts) and I know I had problems due to that that I was trying to go with an action name exactly 22 chars long
Can't remember whether the engine crashed or the function just was not started, though.

The crash is based on an engine bug:

http://www.opserver.de/ubb7/ubbthreads.php?ubb=showflat&Number=457806#Post457806

So this topic can be closed as solved.