you can combine the advantages of both. most racing games do it this way.

use a physics body for the car chassis. it will handle collisions nicely.

use ray casting for the wheels and do your own wheel physics. there is a famous formula for wheel physics but i forgot its name. it should be easy to find with google though.

edit: http://en.wikipedia.org/wiki/Hans_B._Pacejka

(you can't simulate extremely fast spinning wheels with rigid bodies since the frequency of the simulation limits the possible rotational speed.)