sys_nxalloc gets it's memory from the nexus while sys_malloc doesn't. But I wouldn't recommend sys_naxlloc anyways because you cannot free memory allocated this way. (Or has this been fixed in the meanwhile?)

sys_malloc and malloc should get the memory from the same source, i.e. the operating system. If the operating system refuses to give the engine more memory it will refuse to give you more memory as well. So it's not plausible that you'll get more memory when using malloc.

But I feel not completely safe talking about this. Maybe someone else can confirm my statements? Superku? JustSid?