I think WED has indeed a 500 MB limit, probably motivated by the thought that no sane game developer would ever come close. That was back in 2001. But you can override that limit, you need not set the nexus in WED.

Still, when your game uses more total memory than Windows can provide per process, the first thing to crash is DirectX. That happens right at start when the nexus is too big.