For that kind of thing you'll certainly need to write your own function that cycles through the vertices of two objects. Per polygon can be slow as it is, but the fact that polygons might not even intersect (one object completely within another, as you mentioned) is killer. If both objects are convex or you can reduce them to be composed of convex parts, you could use a simplified version of the Gilbert-Johnson-Keerthi distance algorithm. Even then, you'd want them to be fairly low-poly, assuming you want to perform this calculation every frame.

I have a couple of ideas in the back of my mind that might bear fruit if we know more about what kind of situation you're dealing with, and how much control you have over it.

Jibb


Formerly known as JulzMighty.
I made KarBOOM!