I create a simple hollow box in WED and compile a simple bsp map. When using ent_type(NULL) to display what kind of world it's generated, it reports the map created is a mesh based ABT map (ent_type return 2)... my question is how does the engine build a mesh based BSP map (where ent_type returned is 3)? Does it automatically select a mesh based BSP or ABT option depending on level geometry?

I think I have figured this out ... you need to select build bsp map (not simple bsp), set visibility calculations to off and then compile the map.

Before compiling, you need to make up your mind whether you want a Simple Map - meaning no BSP - or a BSP Map. You can't compile a "Simple BSP Map". When the BSP is not generated or not used, for instance when you have no Pro Edition, an ABT is generated by the engine.

The visibility setting or the Mesh setting have no influence on whether it's a BSP map or not. Meshes should always be enabled for A7, otherwise you'll get a slow, ugly A6 level.