Its about time I gave a little back to this forum, so here goes. If anyone wants to improve on this or provide any feedback, suggestions etc, I'd love to hear about it. I make no claims of being anything other than a rank amateur with regards to my programming skills, so if anyone can suggest improvements go right ahead.
The following scripts and methodology are a follow-up to a previous post about empirically testing a strategy's performance against many iterations of a random strategy with a similar distribution of trades.
http://www.opserver.de/ubb7/ubbthreads.php?ubb=showflat&Number=449348#Post449348The approach presented here aims to answer the question 'what is the probability that my strategy's performance was due to luck?' It assumes that the strategy hasn't been curve-fit to the data; consequently I think that the approach is best used by comparing the strategy's out of sample trades, or the results of a walk-forward test.
The idea is that a strategy that you want to trade with real money should outperform a random strategy with the same trade characteristics (number of long/short trades, average trade duration). In fact, you would hope that your real strategy outperformed the vast majority of random strategies, which would strengthen your confidence that you had actually found an exploitable edge.
I think this method has real value when using portfolio strategiess that "cherry pick" certain assets, algorithms and trade directions. For example (and this was the motivation for developing this), the strategy I am working on includes four algorithms. I tested it on 20+ assets in short and long directions separately. I picked the combinations that performed best in a walk forward analysis. Together, and excluding the poor performers, the combination looks like a really good portfolio. But is it really, or have I just picked the combinations that happened to do well in the test period? I hope that this analysis will give me some clues.
For now I'll present the scripts and the methodology, and after I run the tests I will present my results. I'll present the following in this post:
1. A random strategy that mirrors the trade distribution of the real strategy
2. A Unix script for running the random strategy multiple times and outputting performance metrics to text files
3. A method for determining the probability that your strategy beats a random strategy using spreadsheet software.
Firstly, here is the code to generate random trades whose distribution mirrors your actual strategy's distribution. Several inputs are required; these are indicated in the script:
//Randomly generate entries to match a strategy's simulation period, number of trades and trade duration
#define ASSETLOOP while(asset(loop("EUR/USD", "AUD/CHF", "AUD/NZD", "CAD/CHF", "EUR/CHF", "EUR/GBP", "EUR/JPY", "GBP/CAD", "GBP/JPY", "GBP/NZD", "GBP/USD", "NZD/JPY", "NZD/USD", "USD/CAD", "USD/CHF")))
#define TradesLong AlgoVar[0]
#define TradesShort AlgoVar[1]
function getOpt(string param)
{
switch (Asset)
{
case "AUD/CHF":
switch (param)
{
case "strategyTradesLong" : return 64;
case "strategyTradesShort" : return 81;
case "strategyDuration" : return 4;
}
case "AUD/NZD":
switch (param)
{
case "strategyTradesLong" : return 0;
case "strategyTradesShort" : return 264;
case "strategyDuration" : return 4;
}
case "CAD/CHF":
switch (param)
{
case "strategyTradesLong" : return 0;
case "strategyTradesShort" : return 63;
case "strategyDuration" : return 4;
}
case "EUR/CHF":
switch (param)
{
case "strategyTradesLong" : return 0;
case "strategyTradesShort" : return 283;
case "strategyDuration" : return 4;
}
case "EUR/GBP":
switch (param)
{
case "strategyTradesLong" : return 0;
case "strategyTradesShort" : return 220;
case "strategyDuration" : return 4;
}
case "EUR/JPY":
switch (param)
{
case "strategyTradesLong" : return 125;
case "strategyTradesShort" : return 0;
case "strategyDuration" : return 4;
}
case "EUR/USD":
switch (param)
{
case "strategyTradesLong" : return 120;
case "strategyTradesShort" : return 183;
case "strategyDuration" : return 4;
}
case "GBP/CAD":
switch (param)
{
case "strategyTradesLong" : return 102;
case "strategyTradesShort" : return 0;
case "strategyDuration" : return 4;
}
case "GBP/JPY":
switch (param)
{
case "strategyTradesLong" : return 0;
case "strategyTradesShort" : return 173;
case "strategyDuration" : return 4;
}
case "GBP/NZD":
switch (param)
{
case "strategyTradesLong" : return 61;
case "strategyTradesShort" : return 68;
case "strategyDuration" : return 4;
}
case "GBP/USD":
switch (param)
{
case "strategyTradesLong" : return 295;
case "strategyTradesShort" : return 323;
case "strategyDuration" : return 4;
}
case "NZD/JPY":
switch (param)
{
case "strategyTradesLong" : return 291;
case "strategyTradesShort" : return 0;
case "strategyDuration" : return 4;
}
case "NZD/USD":
switch (param)
{
case "strategyTradesLong" : return 71;
case "strategyTradesShort" : return 0;
case "strategyDuration" : return 4;
}
case "USD/CAD":
switch (param)
{
case "strategyTradesLong" : return 0;
case "strategyTradesShort" : return 281;
case "strategyDuration" : return 4;
}
case "USD/CHF":
switch (param)
{
case "strategyTradesLong" : return 258;
case "strategyTradesShort" : return 0;
case "strategyDuration" : return 4;
}
}
}
function tradeRandom()
{
var StrategyTradesLong = getOpt("strategyTradesLong"); //Number of trades in the actual strategy
var StrategyTradesShort = getOpt("strategyTradesShort");
int StrategyDuration = getOpt("strategyDuration"); //Average trade length in the strategy, from the performance report
ExitTime = StrategyDuration;
var NumOpen = NumOpenLong + NumOpenShort;
if (random() > 0) { //so that trades aren't taken on the same bars each run
if(NumOpen == 0 and TradesLong < StrategyTradesLong and random() > 0) { // (NumBars/StrategyTrades) >= 1) {
enterLong();
TradesLong += 1;
}
if (NumOpen == 0 and TradesShort < StrategyTradesShort and random() < 0) {
enterShort();
TradesShort += 1;
}
}
// print(TO_LOG, "TradesLong: %.d", TradesLong);
}
function run()
{
set(LOGFILE);
BarPeriod = 1440;
StartDate = 2010; //Set start and end dates to match those from the acutal strategy's simulation
EndDate = 2014;
LookBack = 0;
if (is(INITRUN)) TradesLong = 0;
if (is(INITRUN)) TradesShort = 0;
ASSETLOOP
while(algo(loop("RAND")))
{
if(Algo == "RAND")
tradeRandom();
}
}
Secondly, here is a Unix code for running the script many times and taking certain performance metrics from the performance report and appending them to separate TXT files for each asset. I use Cygwin to run this script. Obviously, specify your directory path to your Zorro installation. I am running a portfolio test here, but you may need to use Zorro's asset flag if you are not. You can output whatever performance metrics are of interest. Here, I have specified the line containing the profit factor and number of winning/losing trades for each asset. But you can specify whatever parts of the performance report you want to output.
Remember that if you run this script more than once, you will continue to append rows to the existing text file, so be sure to clean up after yourself!
#!/bin/bash
i=0
while [ $i -lt 500 ]
do
cd "C:/Path/Zorro"
./Zorro -run RandomTrades #-a AUD/CHF
cd "C:/Path/Zorro/Log"
sed -n 53p RandomTrades.txt >> RandomTrades_AUDCHF.txt
sed -n 54p RandomTrades.txt >> RandomTrades_AUDNZD.txt
sed -n 55p RandomTrades.txt >> RandomTrades_CADCHF.txt
sed -n 56p RandomTrades.txt >> RandomTrades_EURCHF.txt
sed -n 57p RandomTrades.txt >> RandomTrades_EURGBP.txt
sed -n 58p RandomTrades.txt >> RandomTrades_EURJPY.txt
sed -n 59p RandomTrades.txt >> RandomTrades_EURUSD.txt
sed -n 60p RandomTrades.txt >> RandomTrades_GBPCAD.txt
sed -n 61p RandomTrades.txt >> RandomTrades_GBPJPY.txt
sed -n 62p RandomTrades.txt >> RandomTrades_GBPNZD.txt
sed -n 63p RandomTrades.txt >> RandomTrades_GBPUSD.txt
sed -n 64p RandomTrades.txt >> RandomTrades_NZDJPY.txt
sed -n 65p RandomTrades.txt >> RandomTrades_NZDUSD.txt
sed -n 66p RandomTrades.txt >> RandomTrades_USDCAD.txt
sed -n 67p RandomTrades.txt >> RandomTrades_USDCHF.txt
i=`expr $i + 1`
done
Thirdly, here is a method for estimating the probability that your strategy is better than random:
Having run the random trade generator a large number of times (here I used 500, but I will try running 10,000 overnight and compare results), import the performance metric of interest from its text file to your spreadsheet software (I am using Exel) and construct a histogram and cumulative probability distribution. I used Exel's histogram tool under the Data Analysis function and added a line in the chart for the real strategy's performance metric so that is easy to see.
I hope that this is useful to others, and I'd also like to hear what others think of this approach. I'll present my results in due course.
Cheers
