Tips...

1)Use 'video_set' to set the engine to a higher resolution.
2)Use 'video_window' to remove the border/title/etc of the window, giving you more rendering area.
3)'d3d_antialias' set to a high value like 9 will smooth out the scene, and also 'd3d_mipmapping' may also help improve the quality. Other 'd3d_' functions might be of some use.

Then you can use file_for_screen / bmap_for_screen to grab an image of the screen.

That's all I can think of at the moment.