There is no "miracle function" that gets more memory out of your PC.

C/C++ programs use 'new' for implicit and 'malloc' for explicit memory allocation. What they do depends on the implementation, but under Windows both call HeapAlloc. Replacing that with VirtualAlloc would make no sense. You would get even more "Out of memory" messages, due to larger chunks. You normally don't call low level allocation functions anyway - malloc/new are supposed to use the optimal allocation method.

When you still have not enough memory for your game, your customers won't have either - so reducing the textures is indeed the first thing to do. Also replace vertex with bones animation.