the window which appears when i call engine_open (and it opens at least one time)
also i don't want to use static variables except for things that are really static (like the size of structs)

You don't need to open/close the whole engine to change levels. And who said that you have to use static variables?

because i need a way to check if the engine was already opened
or how would you store this state?

Why do you have to do this? Open the engine, load the editor and make a button "open level". When opening the level, load it in the engine instance. Later, unload the level with level_load(NULL) and reload the next with level_load("myCoolLevel.wmb") or a custom function for WMP parsing.

but this isn't the way i want my expansion to work
i just want to associate a file type with my editor, visual studio manages opening, closing, movement, UI, ...
all i need to do is calling visual studio to show me the correct toolbar and properties
so i don't write an editor from scratch but expanding visual studio