a custom file type is nothing else than a file which you´re filling with values using your own specification. you can choose an extension for your filetype and assign your program to it by modifing the registry (this is normally performed by the setup-routine).
with the commands esper mentioned you can read and write informations to the file. depending on the purpose of your program or data, you have to decide how you want to save. the easiest way is to fire all vars into the file. but sometimes it can be more sensefull to serialize objects or writing in INI or XML format (especially if you want to interchange the data with other programs).
so, for the first version, just consider what information do you need in your editor if you want to restore the thing you´ve edited (positions, rotations, etc) and write these things into the file / read them out on loading
