MatPed, thanks for reading my lengthy post.
There are limitations with WFO, oversampling and Monte Carlo, just as there are limitations to the approach outlined in this thread. The idea is to have numerous tools available, recognising that there is no silver bullet. I'll explore my understanding of these limitations below.
This particular tool is inspired by Ernest Chan’s second book, "Algorithmic Trading", and his hypothesis testing approach to estimating statistical significance of a backtest. As I understood from the book, different methods of estimating the statistical significance of a backtest assume a different probability distribution of the performance statistic of interest. Since we can’t know what the actual distribution is, it makes sense to cover as many bases as possible and at least be informed as to the conditions under which our model could fail.
My understanding is that there are broadly two important considerations in backtesting:
1. Prevent bias from creeping into the strategy development process. This has been covered in the Zorro manual and in most of the books in the recommended reading list, but data snooping bias (that is, using too many parameters that are fitted to random patterns) is probably the one we need to be most concerned about (Zorro is designed in such a way that it is very difficult to introduce look-ahead bias).
2. Statistical significance of our (un-biased) backtest.
The approach I detailed above attempts to shed some light on the latter issue.
WFO is a fantastic tool and is probably the best indicator of future strategy performance, if it is run once and only once. But have you ever built and tested a strategy, and then run a WFO only to find that it didn't perform so well? Have you then ever gone back and tweaked a few parameters and run another WFO to get a better result? How about going over multiple iterations of this process? I know I have done exactly this. Suddenly, the out of sample test results are not so out of sample anymore. I guess this is not so much a limitation of WFO, rather it is a reflection on our limitations as systems developers and one of the issues to be aware of in the development process.
Likewise oversampling is a great tool. It can be used to generate more robust parameters and a greater number of trades in the backtest. It can therefore be used to tackle both of the broad categories of backtest issues I mentioned above. The main limitation of oversampling is that it can’t be used on strategies that exploit for example daily bars, where the strategy depends on price at certain specific times.
Finally, Zorro’s Monte Carlo analysis tool's purpose is to provide a probability distribution of the maximum drawdown and annual return statistics obtained from a finite set of trades (the backtest). Briefly (and this is covered in more detail in the manual and the recommended reading list), its main principle is that while a strategy has a non-random outcome on a large enough sample size, the results of individual trades are random. Therefore, the sequence of wins and losses is random. If you had ten losers in a row followed by ten winners in a row, you’d get a much more significant drawdown than if you had a sequence of twenty trades that alternated win-loss-win-loss…etc. Zorro's Monte Carlo simulator randomly selects trades from the backtest (using replacement) and returns the maximum drawdown and annual return statistics. It then repeats the process many times, obtaining many different equity curves, and estimates the probability distribution of the annual return and maximum drawdown from these curves.
Basically what I am suggesting is one method for tackling the statistical significance problem. It is not meant to replace any of Zorro’s (excellent) development tools. Chan recommends two other methods for this as well, both of which I believe can be implemented in Zorro and both of which use a different method for estimating the probability distribution of interest using Chan's hypothesis testing approach. I'll have a go at implementing these as well.
Cheers
