I'm no longer a fan of Sylex3. It's very dependant and not too stable. Try using the New render-to-texture (RTT) DLL that was posted in the User Contributions. It'll take more coding, but it's well worth it

EDIT: Oh, and to answer your question, as far as I know, Sylex3 doesn't support using any video modes other than the default. There's an english translation of the manual floating around here somewhere if you need it...