Preloading assets in the background

Posted By: JibbSmart

Preloading assets in the background - 05/19/11 19:17

G'day!

I have two requests.

It'd be really cool to be able to preload models, sounds, levels, bmaps, and even arbitrary files in the background, over many frames (like how the http_ functions let the game continue while it waits for data). This could lead to huge levels that are loaded dynamically and all sorts of cool stuff. I imagine it could be used something like this (based on the http example code):
Code:
var id = load_asset("massivelyComplexEnt.mdl");
while (!load_ready(id)) wait(1);
if (load_ready(id) == 1) { // load successful?
	levelReady = 1; // global var to notify everyone the next room is ready
	ent_create("massivelyComplexEnt.mdl", nullvector, roomSetupAction);
} else {
	error("BIG MISTAKE!");
}

Unless, of course, is there already a way to safely do this with our own threading? An additional function to check if an asset has already been loaded or is in the process of being loaded would also be fantastic, but that kind of thing we can manage ourselves. Something like:
Originally Posted By: Imaginary Manual
load_status(var id)
Returns information about an asset's place in memory:

2 asset has already been loaded
1 asset is being loaded
0 asset has not been loaded

My second request is an option to not load the level anew when using session_connect or session_open (or any other session_ functions that require it). I guess this has to do with synchronising entities between sessions, but I'm sure I'm not the only one who sets dplay_entrate to -1 and synchronises everything manually with the send_data_ functions. It'd be nice to set a variable or something for these situations so levels don't need to be loaded again, and then use the session_ functions as we please without having to re-initialise levels. If you've played Burnout Paradise, a very cool features was that you could connect to a game and drop out of it without interrupting your driving, since it was all one big world, and all it had to do on the client side was synchronise a few players, and then remove them when you disconnect.

Thanks for reading laugh

Jibb
Posted By: Quad

Re: Preloading assets in the background - 05/19/11 19:45

load_asset(string assetname,bool force_reload); could be better wink
Posted By: Quad

Re: Preloading assets in the background - 05/19/11 19:46

load_status
return value 3 asset was already loaded, use force_reload to force reloading.
Posted By: JibbSmart

Re: Preloading assets in the background - 05/19/11 19:49

Yes ^^ That would be very useful for level editors, for example.

Jibb
Posted By: MasterQ32

Re: Preloading assets in the background - 05/19/11 20:08

i think you can do something like loading assets wink
buffer_add and multithreading with winapi functions...
Posted By: HeelX

Re: Preloading assets in the background - 05/20/11 07:30

Streaming content is one of those big things I can think of which are missing in Gamestudio. This counts for all kinds of stuff: Bitmaps, Models, Levels, Sounds, etc.

One further suggestion: If one would stream a content and creates it with the common functions like ent_create, the streaming will be continued until it is ready and then it will automatically created by the called function.

This way, one could start streaming e.g. the next level, if the player is in the reach of the level exit. So, lets say it is big and loaded by ~70% when the level_load call is done. Then, the level_load call would wait, until the remaining ~30% of the level is loaded and then it will be "activated".

Sounds reasonable, isn't it? smile
Posted By: TechMuc

Re: Preloading assets in the background - 05/20/11 10:02

Streaming of 3DGS content is possible as richi007 already stated..

At least as long as buffer_add is thread-safe.. but this is a thing jcl has to answer..
Posted By: Zapan@work

Re: Preloading assets in the background - 05/20/11 10:22

You should create a dll for this and sell it, Iam sure many people would like to have it!
Posted By: WretchedSid

Re: Preloading assets in the background - 05/20/11 12:05

Ahh, no one needs a dll and taking money for something this trivial brings bad karma (imo)

Here you go, a working version. It calls add_buffer() on the mainthread, so no need to fear anything. Just save it into a file and include it. Done.
Code:
#include <windows.h>
#include <stdio.h>

typedef struct entA_state
{
	char loaded; // 0 = not begun loading, 1 = loading, 2 = loaded, 3 = Added, 200 = Fuckup
	void *buffer;
	long size;
	
	char *file;
	char *filepath;
	
	struct entA_state *next;
} entA_state;

HANDLE entA_mutex;
entA_state *entA_firstState;

DWORD entA_worker(void *unused);

int entA_init()
{
	entA_firstState 	= NULL;
	entA_mutex 			= CreateMutex(NULL, FALSE, NULL);
	
	if(entA_mutex == NULL)
	{
		return 0;	
	}
	
	DWORD threadID;
	
	// TODO: Error checking
	CreateThread(NULL, 0, entA_worker, NULL, 0, &threadID);
	SetThreadPriority(threadID, THREAD_PRIORITY_LOWEST);	
	
	return 1;
}


DWORD entA_loadObject(entA_state *state)
{
	FILE *file = fopen(state->filepath, "rb");
	if(!file)
	{
		// BAD BAD BAD
		WaitForSingleObject(entA_mutex, INFINITE);
		state->loaded = 200;
		ReleaseMutex(entA_mutex);
		
		return 1;
	}
	
	// Get the size of the file
	fseek(file, 0, SEEK_END);
   state->size = ftell(file);
   rewind(file);
	
	// TODO: Error checking
	state->buffer = (void *)malloc(state->size);
	fread(state->buffer, 1, state->size, file);
	
	fclose(file);


	WaitForSingleObject(entA_mutex, INFINITE);
	state->loaded = 2; // We are done, hooray!
	ReleaseMutex(entA_mutex);

	return 1;
}

DWORD entA_worker(void *unused)
{
	while(1)
	{
		WaitForSingleObject(entA_mutex, INFINITE);
		
		entA_state *state = entA_firstState;
		while(state)
		{
			if(state->loaded == 0)
			{
				state->loaded = 1;
				CreateThread(NULL, 0, entA_loadObject, state, 0, NULL);
			}
			
			state = state->next;
		}
		
		ReleaseMutex(entA_mutex);
	}
	
	return 1;
}

entA_state *entA_stateForFile(char *file, char *filepath)
{
	entA_state *state = entA_firstState;
	while(state)
	{
		if(strcmp(file, state->file) == 0)
		{
			return state;
		}
		
		state = state->next;
	}
	
	state = (entA_state *)malloc(sizeof(entA_state));
	if(state)
	{
		state->file = (char *)malloc((strlen(file) + 1) * sizeof(char));
		state->filepath = (char *)malloc((strlen(filepath) + 1) * sizeof(char));
		
		strcpy(state->file, file);
		strcpy(state->filepath, filepath);
		
		state->loaded 	= 0;
		state->buffer 	= NULL;
		state->next 	= NULL;
		
		state->next = entA_firstState;
		entA_firstState = state;
	}
	
	return state;
}

ENTITY *entA_create(STRING *file, VECTOR *pos, EVENT entAction)
{
	STRING		*filepath;
	ENTITY 		*ent = NULL;
	entA_state 	*state;
	int initiatedLoading = 0;

	filepath = str_create(work_dir);
	str_cat(filepath, "\\");
	str_cat(filepath, file);

	
	WaitForSingleObject(entA_mutex, INFINITE);
	state = entA_stateForFile(_chr(file), _chr(filepath));
	
	if(state->loaded == 3)
	{
		// The file is already cached, so we can safely create the entity right away
		ReleaseMutex(entA_mutex);
		
		ptr_remove(filepath);
		return ent_create(file, pos, entAction);
	}
	else if(state->loaded == 0)
	{
		// We have kicked of the loading, so its our responsibility to add the buffer later on
		initiatedLoading = 1;
	}
	
	ReleaseMutex(entA_mutex);

	int ready = 0;
	while(ready == 0)
	{
		wait(1);
		
		WaitForSingleObject(entA_mutex, INFINITE);
		switch(state->loaded)
		{
			case 2:
			{
				if(initiatedLoading == 1)
				{
					add_buffer(state->file, state->buffer, state->size);
					state->loaded = 3;
					ready = 1;
				}
			}
				break;
				
			case 3:
			{
				ready = 1;
			}
				break;
			
			case 200:
			{
				printf("Could not load file \"%s\"", state->file);
				ReleaseMutex(entA_mutex);
				
				ptr_remove(filepath);
				return NULL;
			}
				break;
			
			default:
				break;
		}
		
		ReleaseMutex(entA_mutex);
	}
	
	ent = ent_create(file, pos, entAction);
	
	ptr_remove(filepath);	
	return ent;
}



There are some error checkings left and I there is no way to purge the files (well they can be purged, but not recreated). Its trivial to add these things, so I let you do this. Here is a small test scrip which uses the models from the shadertest folder:
Code:
#include <acknex.h>
#include "async.c"

ENTITY *stone, *marine;

action rotate()
{
	while(1)
	{
		my.pan += 12 * time_step;
		my.tilt += 9 * time_step;
		
		wait(1);
	}
}

void main()
{
	level_load(NULL);
	wait(1);
	
	entA_init(); // DON'T forget!
	
	
	stone = entA_create("stones.mdl", vector(1024, 0, 0), NULL);
	marine = entA_create("marine.mdl", vector(1024, 0, 0), rotate);
	
	int i;
	for(i=0; i<10; i++)
		entA_create("box.mdl", vector(1024, -290 + (i * 120), -35), NULL);
}


Posted By: jcl

Re: Preloading assets in the background - 05/20/11 12:40

I can confirm that add_buffer is thread safe, and can be used for preloading assets in a separate thread.
Posted By: JibbSmart

Re: Preloading assets in the background - 05/20/11 13:16

Okay, thanks for the info! That's extremely useful!

Any thoughts on session opening/connecting without requiring a level_load when dplay_entrate is -1?

Jibb
Posted By: Dark_samurai

Re: Preloading assets in the background - 05/20/11 13:45

@JustSid: You call something trivial when you need linked lists, a thread that dynamically creates other threads and mutual exclusion ^^ You act like these things are explained in books like "C for dummies" on the second page ^^
Posted By: JibbSmart

Re: Preloading assets in the background - 05/20/11 15:32

Here's another request:
"Thread safe" would be really useful to include in the manual under each thread safe function. That way we can see if any function can be called in a separate thread, and we could even search "thread" in the manual to see all functions that are thread safe.

Jibb
Posted By: WretchedSid

Re: Preloading assets in the background - 05/20/11 20:08

Originally Posted By: Dark_samurai
@JustSid: You call something trivial when you need linked lists, a thread that dynamically creates other threads and mutual exclusion ^^ You act like these things are explained in books like "C for dummies" on the second page ^^

Yes, its more trivial than the concept of wait() which is expected to be understood by everyone here. So yes, I consider this as trivial.

Question: Is anyone interested in a more general purpose solution for the Code that I posted above? So that it works with every engine type loadable from a file and written much cleaner?
Just asking because it looks like no one want this feature in real and everything seems to be about add_buffer() (not even the threadstarter seems to be interested which is imo a bit confusing)
Posted By: JibbSmart

Re: Preloading assets in the background - 05/20/11 20:41

True, add_buffer's thread-safety is of little to no consequence as it's not what makes things slow. I'd be interested in what other functions are thread-safe. Your (JustSid's) example is great, but it'd be nice to be able to just use file_load in another thread, for example (as the add_buffer example in the manual shows).

@jcl:
I know I got carried away when TechMuc said "thread-safe" and jcl said "yes". Any info on the thread-safety of other A8 functions? Any chance this info can be included in the manual?
Originally Posted By: JustSid
Yes, its more trivial than the concept of wait() which is expected to be understood by everyone here. So yes, I consider this as trivial.
No, it's not. I'd bet most users only understand wait as "Leave the function for now and continue from here next frame". Many beginners can get by thinking it's just something you have to put at the end of while(1) loops. Most people here are C-Script/Lite-C programmers, not Windows C programmers.

Jibb
Posted By: WretchedSid

Re: Preloading assets in the background - 05/20/11 20:52

Originally Posted By: JulzMighty
Originally Posted By: JustSid
Yes, its more trivial than the concept of wait() which is expected to be understood by everyone here. So yes, I consider this as trivial.
No, it's not. I'd bet most users only understand wait as "Leave the function for now and continue from here next frame". Many beginners can get by thinking it's just something you have to put at the end of while(1) loops. Most people here are C-Script/Lite-C programmers, not Windows C programmers.

Neither am I a Windows C programmer. A mutual exclusion is much easier to explain than "hey, it returns from the function but in the next frame it resumes the function". And a linked list is the easiest data structure available in C (yes, even simpler than arrays). They make sense in every way while wait() makes no sense to a beginner.

"Hey, you must add a wait(1); to you loop, otherwise your program will crash" and "I'm not sure, have you tried to add a wait(1)?" shows that most people don't really get what wait() is but its assumed to be the magic healing potion for code. wait() is NOT trivial. A linked-list-mutual-exclusion-thread-factory is, compared to wait(), trivial.
Posted By: JibbSmart

Re: Preloading assets in the background - 05/20/11 21:02

wait(1) problems are rare, even on the beginner forums. Please explain to me what a mutual exclusion is -- I can get by with multi-threading, but am still unfamiliar with mutexes. And then explain it and linked lists (which I doubt many Lite-C users try without mastering wait(1) first) and multi-threading more trivially than your own explanation for wait().

Whether or not something is trivial is relative to one's own experience. To say wait() is not trivial gives enough frame of reference to say that implementing your example on one's own is not trivial.

Anyway, whether or not it's trivial is not important. Maybe it is the other way around for someone who's a competent C programmer first before starting with Lite-C. I won't say any more on the matter here. But I'd still like you to explain mutexes to me, please laugh

Jibb
Posted By: JibbSmart

Re: Preloading assets in the background - 05/20/11 21:10

I missed this:
Originally Posted By: JustSid
Just asking because it looks like no one want this feature in real and everything seems to be about add_buffer() (not even the threadstarter seems to be interested which is imo a bit confusing)
Originally Posted By: Jibb (Me)
Okay, thanks for the info! That's extremely useful!
I didn't quote anyone when I wrote that because it was in response to more than one person: You, jcl, Richi007 and TechMuc (I didn't know about add_buffer, and thought when they mentioned it that add_buffer was indeed what needed to be thread-safe, because I assumed it was what loaded the file). I had your example in mind especially when I wrote the latter half. I just didn't name names because it was a thanks to everyone who had contributed towards the idea that this could indeed already be done.

Jibb
Posted By: TechMuc

Re: Preloading assets in the background - 05/21/11 09:06

Quote:
True, add_buffer's thread-safety is of little to no consequence as it's not what makes things slow. I'd be interested in what other functions are thread-safe. Your (JustSid's) example is great, but it'd be nice to be able to just use file_load in another thread, for example (as the add_buffer example in the manual shows).


Just for your undrstanding: File_Load only reads a complete file from harddisk and stores it into the memory.. as long as the method does not contain any strange references to engine-functions they should be thread-safe..

And even if not: Just read the file by yourself via fopen/fread/fclose (methods which are threads safe)
Posted By: WretchedSid

Re: Preloading assets in the background - 05/21/11 09:11

I took the time to write a more generic loader for assets in the background. It uses the same approach as my sample code posted above, but is a bit more cleaner and robust.
http://www.opserver.de/ubb7/ubbthreads.php?ubb=showflat&Number=371433&#Post371433
Posted By: JibbSmart

Re: Preloading assets in the background - 05/21/11 13:41

Quote:
And even if not: Just read the file by yourself via fopen/fread/fclose (methods which are threads safe)
For sure, Sid's example is very helpful with that. And I think you're right that file_load is likely to be thread safe, but I'd like to be sure. Using file_load would do away with the mallocs, fopen, fread, fclose, rewind, and anything else that needs stdio, making an own implementation even more trivial wink

With "inspiration" from Sid's example, this is small enough that it's the kind of thing I'd like to write my own implementation for use in my own future projects.

I guess now that it's the weekend we won't hear more from jcl on this matter until Monday, if ever. My second request was never addressed, and I'd really like to know more about the thread-safety of GS functions -- this'd be very useful to have in the manual. Maybe it'd be even easier to just list the functions that aren't thread-safe, but I don't know.

Jibb
Posted By: jcl

Re: Preloading assets in the background - 05/23/11 11:05

Most engine functions are not thread safe. As a rule of thumb, when an engine function modifies or accesses an engine or level object, it's not thread safe. Functions that play videos and handle file and memory access are thread safe.
Posted By: JibbSmart

Re: Preloading assets in the background - 05/25/11 22:51

Okay, thanks for the info. As it turns out, we can't get the background loading working. There are no errors or crashes, and the entity is created well after the function is first called (depending on how big it is), but there's still a very noticeable pause with large files -- seemingly no different than if we'd just called ent_create without any file loading and buffering first.

Is ent_create doing much behind-the-scenes stuff the first time an entity is created? Can we please have preloading functions so we can prepare an entity without interrupting the game, and then create it (perhaps in a similar way to my example in my first post)? This seems to be an area that A8 is really behind on.

And any comment on my request regarding session_ functions without requiring level_load?

Thanks,

Jibb
Posted By: jcl

Re: Preloading assets in the background - 05/26/11 08:14

Preloading files will not magically remove all the ent_create time. The engine loads files very fast and buffers them in a cache anyway, so you won't gain this much time with a preload process.

Creating entities is a very different process than file loading. Much time is required for DirectX processes, such as building a mesh on the 3D card, building textures, creating the lighting, and creating a collision mesh. This can be sped up a lot by a clever concept that creates entities in several steps.

When you create entities first with INVISIBLE,NOLIGHT,and PASSABLE flags and have collision_mode at 0 at creation time, you'll see that creation is much faster. When you need the entity later, just reset the flags.

You can see such a concept in the infinite terrain demo that creates ten thousands of entities in a second.
Posted By: JibbSmart

Re: Preloading assets in the background - 05/26/11 14:53

Quote:
Preloading files will not magically remove all the ent_create time. The engine loads files very fast and buffers them in a cache anyway, so you won't gain this much time with a preload process.

Creating entities is a very different process than file loading.
Exactly. That's exactly why we can't do this ourselves, and why this discussion is still going on in the Future forum.
Quote:
You can see such a concept in the infinite terrain demo that creates ten thousands of entities in a second.
Yes, because they're all very simple. The number of the same entity is almost inconsequential. In my tests I used a cube with an enormous texture -- in this specific case it's only the texture that causes the delay, and that'll often be the case as these days one entity will often have many skins. If I change my test to ent_create the same model again after the first one is created, there's no delay for the second entity.

It's this first-time-creating, behind-the-scenes stuff that we can't do anything about that could surely be done in the background with an engine function.

Even if "pre-loading" isn't going to do what we want, I thought the intention of this request was made very clear, if not from my first post then from subsequent responses from everyone else.

Jibb
Posted By: jcl

Re: Preloading assets in the background - 05/26/11 15:49

What we're trying to explain to you is that there is no "behind the scenes stuff". All preload features you can have are already there. You're probably the 500th user who asks on the Future forum for a solution to overcome load delays, and over the years we implemented many functions and methods for that. Just look for "preload" in the manual.

Only thing you need to understand is that at some time, textures must be loaded and meshes must be created. You can not prevent this, you can only control when it happens.

Textures don't cause a delay when you create your cube. They only cause a delay when the cube either becomes visible in the level, or when it is preloaded. So it's totally up to you at which time you want this delay. The delay itself can not be charmed away by a 3D engine. Mesh creation and texture loading to the 3D card takes time and happens deep inside the DirectX driver and on your hardware.
Posted By: JibbSmart

Re: Preloading assets in the background - 05/26/11 16:52

Quote:
What we're trying to explain to you is that all preloading functions you can have are already there. You're probably the 500th user who asks on the Future forum for some solution to overcome load delays, and over the years we implemented many functions and methods for that. Just look for "preload" in the manual.
One would think you'd be better at answering the question by now, then. One would also think a search for "ent_create" or "load" in the Future forum would yield examples.
Quote:
Only thing you need to understand is that at some time, textures must be loaded and meshes must be created. You can not prevent this, you can only control when it happens.
This is obvious, and the least relevant piece of information in a thread that is breaking world records for number of mentions of "thread". No one cares how slow an algorithm or function is if it can be threaded. So it's about time you said it can't be.
Quote:
The delay itself can not be affected by a 3D engine - not matter how often you demand this here. It's the texture loading to the 3D card, and happens in the DirectX driver and on your hardware.
This is your first such mention. Seriously, whenever a request is made for something you can do nothing about, say so. Far out. How are we meant to take your answers seriously if you're generally vague, dismissive, or ignoring requests that still haven't been addressed (multiplayer). I understand that the difference between an engine programmer and an engine user means the intention of this discussion wasn't clear to you until a few posts back, but once you get that understanding it makes sense to address the question with "This is done in DirectX and cannot be threaded" instead of losing your cool. From the get go we've said we either want to thread this ourselves or for you to thread it.

Given that it can't be done, I'm sure we'll figure out for ourselves how so many games with huge assets load them without delay.

Jibb

EDIT: Since you edited your cool back into your post, please re-read mine with a laugh at the end of every sentence, and ignore any suggestion that I might be frustrated.

EDIT2: By the way, that's not sarcastic. I really am all chill now having read version 2.0 of your post laugh
© 2024 lite-C Forums