I haven't fiddled with the collision engine for quite a while, but recently I came across this info:

Originally Posted By: Manual: Block Properties
Invisible - Invisible block. When setting the Invisible = None flag in the map compiler dialog, the block is still used in the BSP tree, can clip away objects behind it, and throws static shadows. Otherwise it's completely invisible for the view and the AABB collision detection.

So apparently this compiler setting has to be enabled. However, invisible colliders that drop shadows aren't that great either... Perhaps there's a better solution? I'm interested in this, too.