Could be some memory-leak in the engine itself.
Maybe a bug in some function.
Or the engine does not garbage-collect temporary objects/arrays.
The only chance to find it would be then to turn certain parts of
the code on and off.
Do you use physics or particles, or an enternal dll?
Try out a level-load and see if the memory gets cleared up.
If you cant figure out what the cause is, and its not
some fault in your code (undeleted entities or unexited functions in a loop), you should contact the developers.