If I got you right, matWorldInv is what you search. Thus try to transform your 3d(!) cubemap-coordinates by matWorldInv - that should do the tick.