I attach some words and 1 image from the book that explains better the logic. Do you think that this metod could help to detect overfitting?
[...] The meaning of the trading system’s complexity
Figure 5.11: Finding an optimal rule complexity for system LUXOR for British pound/US dollar (FOREX) training and test period. The system’s input parameters are optimised for maximum total net profit, from left to right, one after another, within the training period 21/10/2002-28/2/2007 (blue line = training results). Then results are checked in test data range 1/3/2007-4/7/2008 (green line = test results).
How can you now interpret this behaviour of the trading system? Let’s start within the diagram (Figure 5.11) from the left side when no system parameter is optimised. There the trading system has a very low complexity, since the applied rules are quite simple and not optimised at all. If you start to optimise the first parameter the system’s performance changes markedly. The raw and simple trading logic used so far can easily be made better. Interestingly when not many optimised parameters have been introduced the behaviour within the test range sometimes changes more than within the training range. The change can be much worse, but it can be better than the improvement that takes place in the training range. The reason for this behaviour is that a system that has only been optimised a little reacts very sensitively to parameter changes because there are not many parameters in place yet. The rule complexity and the predictability for your test set is low. Furthermore, keep in mind that much of what happens in different market phases and areas is accidental and also depends on the market sample bias. It can be that the out-ofsample data period is more “friendly” to our trading system logic in a certain stage than the training data period. With further parameters being optimised or added the changes in the system’s reaction become smaller but still performance improves in the out-ofsample test data range. With the first three parameters being optimised our trading system reaches an important point: it reaches its optimal complexity. From this point on every further optimised parameter (risk stop, trailing stop, profit target) decreases the system’s performance in the test region although the results still improve in the training region. You now have the situation of curve over-fitting. Every new optimised parameter improves the fit of the system to the training area but what happens here is more an adjustment to the existing market noise than an improvement in predictive capability. Thus the net profit within the test region does not become bigger with further optimised parameters but instead it decreases from the fourth parameter onwards. You now again have an out-of-sample deterioration. [...]
