It is unlikely that the resources of a normal game consume all the available memory on a modern PC. When you have no memory left, normally something is wrong either with your game, or with your Windows memory settings.

You can easily find the problem by checking at which moment the error message appears. For instance, when it appears when loading a certain entity, something with that entity might be wrong.