Its about time I gave a little back to this forum, so here goes. If anyone wants to improve on this or provide any feedback, suggestions etc, I'd love to hear about it. I make no claims of being anything other than a rank amateur with regards to my programming skills, so if anyone can suggest improvements go right ahead.
The approach presented here aims to answer the question 'what is the probability that my strategy's performance was due to luck?' It assumes that the strategy hasn't been curve-fit to the data; consequently I think that the approach is best used by comparing the strategy's out of sample trades, or the results of a walk-forward test.
The idea is that a strategy that you want to trade with real money should outperform a random strategy with the same trade characteristics (number of long/short trades, average trade duration). In fact, you would hope that your real strategy outperformed the vast majority of random strategies, which would strengthen your confidence that you had actually found an exploitable edge.
I think this method has real value when using portfolio strategiess that "cherry pick" certain assets, algorithms and trade directions. For example (and this was the motivation for developing this), the strategy I am working on includes four algorithms. I tested it on 20+ assets in short and long directions separately. I picked the combinations that performed best in a walk forward analysis. Together, and excluding the poor performers, the combination looks like a really good portfolio. But is it really, or have I just picked the combinations that happened to do well in the test period? I hope that this analysis will give me some clues.
For now I'll present the scripts and the methodology, and after I run the tests I will present my results. I'll present the following in this post:
1. A random strategy that mirrors the trade distribution of the real strategy 2. A Unix script for running the random strategy multiple times and outputting performance metrics to text files 3. A method for determining the probability that your strategy beats a random strategy using spreadsheet software.
Firstly, here is the code to generate random trades whose distribution mirrors your actual strategy's distribution. Several inputs are required; these are indicated in the script:
Code:
//Randomly generate entries to match a strategy's simulation period, number of trades and trade duration
#define ASSETLOOP while(asset(loop("EUR/USD", "AUD/CHF", "AUD/NZD", "CAD/CHF", "EUR/CHF", "EUR/GBP", "EUR/JPY", "GBP/CAD", "GBP/JPY", "GBP/NZD", "GBP/USD", "NZD/JPY", "NZD/USD", "USD/CAD", "USD/CHF")))
#define TradesLong AlgoVar[0]
#define TradesShort AlgoVar[1]
function getOpt(string param)
{
switch (Asset)
{
case "AUD/CHF":
switch (param)
{
case "strategyTradesLong" : return 64;
case "strategyTradesShort" : return 81;
case "strategyDuration" : return 4;
}
case "AUD/NZD":
switch (param)
{
case "strategyTradesLong" : return 0;
case "strategyTradesShort" : return 264;
case "strategyDuration" : return 4;
}
case "CAD/CHF":
switch (param)
{
case "strategyTradesLong" : return 0;
case "strategyTradesShort" : return 63;
case "strategyDuration" : return 4;
}
case "EUR/CHF":
switch (param)
{
case "strategyTradesLong" : return 0;
case "strategyTradesShort" : return 283;
case "strategyDuration" : return 4;
}
case "EUR/GBP":
switch (param)
{
case "strategyTradesLong" : return 0;
case "strategyTradesShort" : return 220;
case "strategyDuration" : return 4;
}
case "EUR/JPY":
switch (param)
{
case "strategyTradesLong" : return 125;
case "strategyTradesShort" : return 0;
case "strategyDuration" : return 4;
}
case "EUR/USD":
switch (param)
{
case "strategyTradesLong" : return 120;
case "strategyTradesShort" : return 183;
case "strategyDuration" : return 4;
}
case "GBP/CAD":
switch (param)
{
case "strategyTradesLong" : return 102;
case "strategyTradesShort" : return 0;
case "strategyDuration" : return 4;
}
case "GBP/JPY":
switch (param)
{
case "strategyTradesLong" : return 0;
case "strategyTradesShort" : return 173;
case "strategyDuration" : return 4;
}
case "GBP/NZD":
switch (param)
{
case "strategyTradesLong" : return 61;
case "strategyTradesShort" : return 68;
case "strategyDuration" : return 4;
}
case "GBP/USD":
switch (param)
{
case "strategyTradesLong" : return 295;
case "strategyTradesShort" : return 323;
case "strategyDuration" : return 4;
}
case "NZD/JPY":
switch (param)
{
case "strategyTradesLong" : return 291;
case "strategyTradesShort" : return 0;
case "strategyDuration" : return 4;
}
case "NZD/USD":
switch (param)
{
case "strategyTradesLong" : return 71;
case "strategyTradesShort" : return 0;
case "strategyDuration" : return 4;
}
case "USD/CAD":
switch (param)
{
case "strategyTradesLong" : return 0;
case "strategyTradesShort" : return 281;
case "strategyDuration" : return 4;
}
case "USD/CHF":
switch (param)
{
case "strategyTradesLong" : return 258;
case "strategyTradesShort" : return 0;
case "strategyDuration" : return 4;
}
}
}
function tradeRandom()
{
var StrategyTradesLong = getOpt("strategyTradesLong"); //Number of trades in the actual strategy
var StrategyTradesShort = getOpt("strategyTradesShort");
int StrategyDuration = getOpt("strategyDuration"); //Average trade length in the strategy, from the performance report
ExitTime = StrategyDuration;
var NumOpen = NumOpenLong + NumOpenShort;
if (random() > 0) { //so that trades aren't taken on the same bars each run
if(NumOpen == 0 and TradesLong < StrategyTradesLong and random() > 0) { // (NumBars/StrategyTrades) >= 1) {
enterLong();
TradesLong += 1;
}
if (NumOpen == 0 and TradesShort < StrategyTradesShort and random() < 0) {
enterShort();
TradesShort += 1;
}
}
// print(TO_LOG, "TradesLong: %.d", TradesLong);
}
function run()
{
set(LOGFILE);
BarPeriod = 1440;
StartDate = 2010; //Set start and end dates to match those from the acutal strategy's simulation
EndDate = 2014;
LookBack = 0;
if (is(INITRUN)) TradesLong = 0;
if (is(INITRUN)) TradesShort = 0;
ASSETLOOP
while(algo(loop("RAND")))
{
if(Algo == "RAND")
tradeRandom();
}
}
Secondly, here is a Unix code for running the script many times and taking certain performance metrics from the performance report and appending them to separate TXT files for each asset. I use Cygwin to run this script. Obviously, specify your directory path to your Zorro installation. I am running a portfolio test here, but you may need to use Zorro's asset flag if you are not. You can output whatever performance metrics are of interest. Here, I have specified the line containing the profit factor and number of winning/losing trades for each asset. But you can specify whatever parts of the performance report you want to output.
Remember that if you run this script more than once, you will continue to append rows to the existing text file, so be sure to clean up after yourself!
Code:
#!/bin/bash
i=0
while [ $i -lt 500 ]
do
cd "C:/Path/Zorro"
./Zorro -run RandomTrades #-a AUD/CHF
cd "C:/Path/Zorro/Log"
sed -n 53p RandomTrades.txt >> RandomTrades_AUDCHF.txt
sed -n 54p RandomTrades.txt >> RandomTrades_AUDNZD.txt
sed -n 55p RandomTrades.txt >> RandomTrades_CADCHF.txt
sed -n 56p RandomTrades.txt >> RandomTrades_EURCHF.txt
sed -n 57p RandomTrades.txt >> RandomTrades_EURGBP.txt
sed -n 58p RandomTrades.txt >> RandomTrades_EURJPY.txt
sed -n 59p RandomTrades.txt >> RandomTrades_EURUSD.txt
sed -n 60p RandomTrades.txt >> RandomTrades_GBPCAD.txt
sed -n 61p RandomTrades.txt >> RandomTrades_GBPJPY.txt
sed -n 62p RandomTrades.txt >> RandomTrades_GBPNZD.txt
sed -n 63p RandomTrades.txt >> RandomTrades_GBPUSD.txt
sed -n 64p RandomTrades.txt >> RandomTrades_NZDJPY.txt
sed -n 65p RandomTrades.txt >> RandomTrades_NZDUSD.txt
sed -n 66p RandomTrades.txt >> RandomTrades_USDCAD.txt
sed -n 67p RandomTrades.txt >> RandomTrades_USDCHF.txt
i=`expr $i + 1`
done
Thirdly, here is a method for estimating the probability that your strategy is better than random:
Having run the random trade generator a large number of times (here I used 500, but I will try running 10,000 overnight and compare results), import the performance metric of interest from its text file to your spreadsheet software (I am using Exel) and construct a histogram and cumulative probability distribution. I used Exel's histogram tool under the Data Analysis function and added a line in the chart for the real strategy's performance metric so that is easy to see.
I hope that this is useful to others, and I'd also like to hear what others think of this approach. I'll present my results in due course.
I do not if I have fully understood the logic behind your work, but thank you to have shared your ideas with us. My main question is: why do you think that wfo, oversampling and montecarlo are not enough to avoid the "luck" factor?
In particular Zorro's test give a confidence statistics that should be read something like: you have 75% (or whatever) probability to gain X, with Y capital i.e. you can expect to be 75% of times lucky...
I will be happy to over-test your strategy if you share it with us and.
Witch asset did you use? Can share the history data of this assets with us?
Ciao
Re: Testing whether a strategy is better than random
[Re: MatPed]
#449599 03/24/1523:1003/24/1523:10
There are limitations with WFO, oversampling and Monte Carlo, just as there are limitations to the approach outlined in this thread. The idea is to have numerous tools available, recognising that there is no silver bullet. I'll explore my understanding of these limitations below.
This particular tool is inspired by Ernest Chan’s second book, "Algorithmic Trading", and his hypothesis testing approach to estimating statistical significance of a backtest. As I understood from the book, different methods of estimating the statistical significance of a backtest assume a different probability distribution of the performance statistic of interest. Since we can’t know what the actual distribution is, it makes sense to cover as many bases as possible and at least be informed as to the conditions under which our model could fail.
My understanding is that there are broadly two important considerations in backtesting:
1. Prevent bias from creeping into the strategy development process. This has been covered in the Zorro manual and in most of the books in the recommended reading list, but data snooping bias (that is, using too many parameters that are fitted to random patterns) is probably the one we need to be most concerned about (Zorro is designed in such a way that it is very difficult to introduce look-ahead bias). 2. Statistical significance of our (un-biased) backtest.
The approach I detailed above attempts to shed some light on the latter issue.
WFO is a fantastic tool and is probably the best indicator of future strategy performance, if it is run once and only once. But have you ever built and tested a strategy, and then run a WFO only to find that it didn't perform so well? Have you then ever gone back and tweaked a few parameters and run another WFO to get a better result? How about going over multiple iterations of this process? I know I have done exactly this. Suddenly, the out of sample test results are not so out of sample anymore. I guess this is not so much a limitation of WFO, rather it is a reflection on our limitations as systems developers and one of the issues to be aware of in the development process.
Likewise oversampling is a great tool. It can be used to generate more robust parameters and a greater number of trades in the backtest. It can therefore be used to tackle both of the broad categories of backtest issues I mentioned above. The main limitation of oversampling is that it can’t be used on strategies that exploit for example daily bars, where the strategy depends on price at certain specific times.
Finally, Zorro’s Monte Carlo analysis tool's purpose is to provide a probability distribution of the maximum drawdown and annual return statistics obtained from a finite set of trades (the backtest). Briefly (and this is covered in more detail in the manual and the recommended reading list), its main principle is that while a strategy has a non-random outcome on a large enough sample size, the results of individual trades are random. Therefore, the sequence of wins and losses is random. If you had ten losers in a row followed by ten winners in a row, you’d get a much more significant drawdown than if you had a sequence of twenty trades that alternated win-loss-win-loss…etc. Zorro's Monte Carlo simulator randomly selects trades from the backtest (using replacement) and returns the maximum drawdown and annual return statistics. It then repeats the process many times, obtaining many different equity curves, and estimates the probability distribution of the annual return and maximum drawdown from these curves.
Basically what I am suggesting is one method for tackling the statistical significance problem. It is not meant to replace any of Zorro’s (excellent) development tools. Chan recommends two other methods for this as well, both of which I believe can be implemented in Zorro and both of which use a different method for estimating the probability distribution of interest using Chan's hypothesis testing approach. I'll have a go at implementing these as well.
Witch asset did you use? Can share the history data of this assets with us?
Ciao
I tested the strategy on all the currency pairs available from www.histdata.com.
I would be more than happy to share my history with you. However, it is probably easier if you just get it from the website I mentioned above, which provides the data at a small cost to cover their server costs.
I got your points. I understand you are far in front of me in the algorithmic trading learning curve too. Can you just summarize the minus/plus of Chan approach vs the "traditional" zorro's?
Regarding the data I was looking for a shortcut in the conversion process. I already have all the data I need from tickhistory. Its just a boring process split data in a year file, apply the script,...
The integration between a Data Provider and Zorro is one of the items in my Zorro's wish list (as amibroker or multicharts)
Anyway if you do not want to share then for any ethical reason, I understand that.
Ciao
Last edited by MatPed; 03/25/1501:49.
Re: Testing whether a strategy is better than random
[Re: boatman]
#449603 03/25/1501:5903/25/1501:59
In this post I'll present the results from testing my strategy using the approach described in the first post. Right at the outset, I'll say that the results presented here are flawed in that they consider the aggregated performance of the four algorithms on each market. It occurred to me as I was compiling the results that it would make more sense to analyse each algorithm on each market separately. I'll do this in the future, but for now this is a decent starting point and hopefully illustrates how this approach works and that it has some value.
The strategy is simple enough - it attempts to exploit short term serial correlation (or lack thereof) in an asset's return series. There are four algos: Algo 1 assumes that short term autocorrelation is persistent and buys or sells in the prevailing direction if the autocorrelation coefficient rises above a certain threshold. Algo 2 assumes that autocorrelation is not persistent and enters in the opposite direction if the autocorrelation coefficient rises above the threshold. Algos 3 and 4 use the same approach but use anti-autocorrelation. The algos are switched on or off via an equity curve approach as described in the manual. The correlation coefficient lag and lookback parameters are optimized.
I tested it on a bunch of assets and picked the algo/asset/direction combinations that produced good results in the WFO (I realise that this introduces a form of bias into the development process).
Which combinations were profitable due to the strategy's predictive power, and which were profitable simple due to the bias in the market during the test period? I used the approach described in the first post to try to answer that question.
The attached spreadsheet shows the results of the backtest selecting the combinations that performed well in the WFO (first tab). The equity curve looks great, as you would expect due to the bias introduced into the development process. There are a few additional performance statistics that I look at included in the spreadsheet if anyone is interested.
The second tab shows the results from the random test. The test statistic selected is profit factor. This sheet includes a histogram and cumulative frequency chart for the profit factor obtained for each asset in the random test (10,000 iterations). Lets select a cumulative frequency limit of 90% for selecting assets. That is, for an asset to be included in our strategy, the profit factor generated in the backtest must be better than 90% of the profit factors generated in the random test. Anything less than this indicates that the results obtained were possibly due to the nature of the market during the simulation time (for example, an asset traded only on the long side will do well in a rising market, even if entries are random). The 90% value is arbitrary; the value selected would depend on the individual. Perhaps 95% is a better choice.
Interestingly, some of the assets returned what seems a very good profit factor, but didn't beat an overwhelming majority of the random strategies. Take a look at the results for EUR/CHF for example, with a PF of 1.39 returned by the strategy. This only beat about 75% of the random strategies, which indicates that there is a good chance that the strategy was not exploiting any tradeable edge; rather it did well simply by being in the market in the right direction. Indeed, the strategy only selected this pair for trading short, and the results appear to be largely due to this bias.
Another question that occurred to me while I was compiling the results is whether all this testing could be deemed unnecessary if my strategy development process were more elegant, introducing as little bias as possible? Or is it appropriate to take a 'brute force' approach, testing as I have in this case and potentially introducing bias, only to strip away the flawed combinations using methods such as this? In the latter approach, you would be casting the net far and wide so to speak, hoping to catch something of value and eliminating the rest. The former approach, in my mind, risks missing a strategy of value, but ensures that anything that shows promise actually shows real promise and not something ephemeral.
I hope this kicks off some healthy discussion. I'll run another test on individual asset/algorithm combinations overnight and post the results in due course.
Cheers
Edit: spreadsheet too big to upload, so attached instead is a screenshot of the results.
Updated Unix code below. This one outputs all the desired data from each iteration of the random strategy to one text file. The idea is to use Exel's text import and data sort functions rather than outputting each asset/algo combination to a separate file.
Code:
#!/bin/bash
i=0
while [ $i -lt 3 ]
do
cd "C:/Path/Zorro"
./Zorro -run RandomTrades
cd "C:/Path/Zorro/Log"
sed -n 71,115p RandomTrades.txt >> RandomTrades_AllCombos.txt
i=`expr $i + 1`
done
Can you just summarize the minus/plus of Chan approach vs the "traditional" zorro's?
Please don't get the impression that this is a 'one or the other' type of situation where we choose one approach over another. The approach above has merit and limitations, as do Chan's other approaches and Zorro's tools. The idea is to understand how the different tools work and what their limitations are, and then applying them in a logical manner. Remember that we are dealing with uncertainty, and no single approach or combination of approaches will yield a conclusive solution.
I believe I've outlined above what to me are the main limitations of some of Zorro's tools that relate to backtesting and statistical significance. Chan uses a hypothesis testing approach in which we hypothesise that the true test statistic of interest, based on an infinite data set (not a finite backtest) is zero (or 1, presumably, if we are looking at profit factor). Using this approach, we aim to reject this hypothesis with a certain degree of confidence. The probability distribution of the test statistic must have a zero mean, if our hypothesis is true. Therefore, if we know (or assume) a probability distribution, we can compute the probability that our statistic will be as large as that returned by the backtest. Obviously, the smaller this probability, the more confidently we can reject the hypothesis.
The main issue with this approach is determining the probability distribution of our statistic under the hypothesis we hope to reject. Hence, Chan's three approaches. The one I detailed is essentially an attempt to empirically model this probability distribution and see where the strategy's backtested statistic is located upon it.
There are other limitations which Chan cites in his book too. For example, the fact that the hypothesis we hope to reject isn't unique, and different hypotheses can give rise to different estimates of statistical significance.
Originally Posted By: MatPed
Regarding the data I was looking for a shortcut in the conversion process. I already have all the data I need from tickhistory. Its just a boring process split data in a year file, apply the script,...
The integration between a Data Provider and Zorro is one of the items in my Zorro's wish list (as amibroker or multicharts)
Anyway if you do not want to share then for any ethical reason, I understand that.
Well no one ever accused me of being ethical! The reason I suggested paying for the data is for simplicity. For about $30, you can transfer gigabytes of data using an FTP client. I don't know how to set this up so that you can access the data in a similar fashion from my PC, and to be honest I've got better things to do than try to save you thirty bucks. Having said that, if you have a Dropbox account with enough storage (>50 GB) I could quite easily share it with you that way. I'd prefer to get permission from the www.histdata.com site owner first though.
If you do manage to get your hands on the data, I have some scripts that automate the conversion process that I would be happy to share. One that I paid someone on Elance to create, and the credit for the other one goes to a fellow Zorro user (yosoytrader). I assume he wouldn't be against me sharing his script, but again, I'd prefer to check first.
You have been clear, thank you. Regarding the data, I have all the space you required, nut I do not want to create any issue, if you prefer to give the scripts only, I will handle the data, If you want to give the data already transformed in the Zorro's format great. Your call! Anyway I will appreciate your support.
Please bear in mind that English is not my native language. Some sentence or words picking, may not be appropriate!
Re: Testing whether a strategy is better than random
[Re: MatPed]
#449677 03/28/1506:1503/28/1506:15
