I am thinking you have a problem with your geometry..

On the walls to the left and right are divided into 4 parts(the big "X" shape) make sure there is a vertex in the center, other wise you have overlaping and possibly problems with normals....also move your "ball" to the opposite side of where you have a collision issue and look at the collision area(it may be a polygon with a flipped normal)...and of course check your normals.
Are you setting the polygon flags(polly collision)if not it can also be th bounding box you are colliding with....?