Regarding math and vector / matrix calculation. Actually sometimes it is even better and faster to setup your own routines. I tell you why:

If you want to do translation, rotation and scaling within the same frame then you have to use 3 operations. If you do that on a mesh with 100 thousand vertices, then you get 3 times the amount of operations.
But if you create a combo matrix / vector first, that combines translation, scaling and rotation in one matrix / vector, then you have only 1/3 the amount of operations, resulting in higher speed.

But you dont need to write your own engine only because of this. It is just some kind of optimization.