A very good reference:
Meyer, Carl. (2000). Matrix analysis and applied linear algebra. Published by the Society for Industrial and Applied Maths (SIAM). Meyer is pretty definitive for matrices. If you peruse any of the standard computer graphics textbooks (e.g. Foley, Watt, etc.) you'll notice that a few topics recur (1) Homogenous coordinate systems, (2) matrices including inversion-rotation-transformation, and (3) vectors. You can throw in a big bundle of partial differential equation theory as well (continuous functions and friends)- cf. Calculus and Analytical Geometry. Unfortunately, I know of no book entitled 'Matrices without tears'.