I have that book too, but what they wrote in this section is in fact wrong.

A system can normally not deteriorate by overfitting parameters, only its test results get more inaccurate. This is not intuitive, but is mathematically true under normal conditions. A system with the highest profit in a certain period has also the highest profit expectancy in another period. This does not mean of course that it is profitable at all.

This is also true for the Luxor system from the book. The green performance peak in the image at "optimal complexity" is in fact more overfitted than the performance of "Rules too complex". Reason is that the whole system was adapted to the historical period of the book. Thus, adding or optimizing rules with in sample data simply reduces the system's fit to the out of sample data. That's why you see lower performance there, not because of some overfitting effect.

The Luxor system rules were carefully selected to achieve the maximum return with the data period of the book. The system has no out of sample period at all. You should see all its results under that aspect.