Can you please help me figure out what's wrong with this. As a n00b, I'm trying to understand the major pitfalls that can lead to this type of too-optimistic result.
Code:
Walk-Forward Test: Workshop6_2_dusktrader portfolio 2002..2013
Read Workshop6_2_dusktrader.fac Workshop6_2_dusktrader_1.par Workshop6_2_dusktrader_2.par Workshop6_2_dusktrader_3.par Workshop6_2_dusktrader_4.par Workshop6_2_dusktrader_5.par Workshop6_2_dusktrader_6.par Workshop6_2_dusktrader_7.par
Profit 2141744$ MI 28546$ DD 1$ Capital 9180$
Trades 1241 Win 61% Avg +173291.9p Bars 761
AR 685102% PF 539.68 SR 11.16 UI 0.1% Error 22%
Generate Chart - please wait... ok
In studying Workshop 6_2, I decided to make a few changes. First, I didn't like the idea that the margin calculation was being lumped together for all the trade types, because they weren't all profitable. Also, I wanted to improve overall robustness by adding all the other pairs available to me.
So I (attempted to make) these changes: * Only trade if OptimalF result was positive * Don't trade any margin for now (that skews results) * Test all available pairs, as much history as possible, in NFA mode
Questions I have now: * Is it possible/correct for some pairs to have an OptimalF value of .999? That happens if I test over a shorter period. * Is it possible/correct for some pairs to have such high profit factors as shown here? * Is it true that when history data is not available, that there would be no negative effect on the backtest? For example, I requested backtesting to 2002, but some pairs do not have that much history. My assumption is that it will just skip those pairs if no data available.
THANKS
Code:
// Workshop 6: Portfolio trading ///////////////////
function tradeTrend()
{
vars Price = series(price());
vars Trend = series(LowPass(Price,optimize(250,100,1000)));
Stop = optimize(4,2,8) * ATR(100);
Trail = 0;
if(valley(Trend) && OptimalFLong > 0) //long setup, known profitable
{
// reinvest the square root of profits
var MarginLong = sqrt(WinLong-LossLong)*OptimalFLong;
Margin = clamp(MarginLong, 5, 100); //max $1000 per trade
enterLong();
}
else if(peak(Trend) && OptimalFShort > 0) //short setup, known profitable
{
// reinvest the square root of profits
var MarginShort = sqrt(WinShort-LossShort)*OptimalFShort;
Margin = clamp(MarginShort, 5, 100); //max $1000 per trade
enterShort();
}
}
function tradeCounterTrend()
{
vars Price = series(price());
vars DomPeriod = series(DominantPeriod(Price,30));
var LowPeriod = LowPass(DomPeriod,500);
vars HP = series(HighPass(Price,LowPeriod*optimize(1,0.5,2)));
vars Signal = series(Fisher(HP,500));
var Threshold = optimize(1,0.5,2,0.1);
Stop = optimize(4,2,8) * ATR(100);
Trail = 4*ATR(100);
if(crossUnder(Signal,-Threshold) && OptimalFLong > 0) //long setup, known profitable
{
// reinvest the square root of profits
var MarginLong = sqrt(WinLong-LossLong)*OptimalFLong;
Margin = clamp(MarginLong, 5, 100); //max $1000 per trade
enterLong();
}
else if(crossOver(Signal,Threshold) && OptimalFShort > 0) //short setup, known profitable
{
// reinvest the square root of profits
var MarginShort = sqrt(WinShort-LossShort)*OptimalFShort;
Margin = clamp(MarginShort, 5, 100); //max $1000 per trade
enterShort();
}
}
function run()
{
set(PARAMETERS+FACTORS+NFA); // generate and use optimized parameters
BarPeriod = 240; // 4 hour bars
StartDate = 2002;
LookBack = 500; // needed for Fisher()
NumWFOCycles = 8; // activate WFO
//if(ReTrain) {
// UpdateDays = -1; // update price data from the server
// SelectWFO = -1; // select the last cycle for re-optimization
//}
// portfolio loop
while(asset(loop("EURUSD","USDJPY","AUDUSD","EURCHF","GBPUSD","NZDUSD","USDCAD","USDCHF")))
while(algo(loop("TRND","CNTR")))
{
// initiate trade if known profitable track record
if(OptimalFLong > 0 or OptimalFShort > 0)
{
if(strstr(Algo,"TRND"))
tradeTrend();
else if(strstr(Algo,"CNTR"))
tradeCounterTrend();
}
}
PlotWidth = 800;
PlotHeight1 = 320;
}
EDIT: I made some improvements to the script by allowing it to set margin as the square root of equity gain (I think I did this right), clamping trades to $1000 max margin. I updated the script and stats above.
Last edited by dusktrader; 07/22/1315:57.
Re: Too good to be true? Help me poke holes in it please
[Re: dusktrader]
#426376 07/22/1316:0607/22/1316:06
Okay, first of all, you can get an OptimalF of 0.999 if all your trades for that algo were profitable (Already happened to me, too).
To your results: This is from the manual (http://zorro-trader.com/manual/en/tutorial_kelly.htm): "The OptimalF factors are not calculated in every WFO step, but for the whole period" Now, as you only trade algos with a positive OptimalF, you only trade the algos that have already proven to be profitable over the whole simulation period. So you would definitely get worse results in real trading.
But thats only my opinion. And it's kind of strange as Zorro would have to know the final OptimalF-factor while the test is still running. EDIT: It's not strange, the factors are created while training. I'm not very familiar with the calculations of the OptimalF-factors.
I just looked at your picture. Do you notice that many of your trades (all profitable) stay open till the end? This is never a good sign as these trades got closed because your simulation period was over. It does not prove any reliability of your exit conditions. Most likely you just were lucky regarding the overall trend of the asset.
Last edited by blaub4r; 07/22/1317:53.
Re: Too good to be true? Help me poke holes in it please
[Re: blaub4r]
#426378 07/22/1316:4607/22/1316:46
Thanks blaub4r. I'm attaching here also EURUSD's result chart, one of the worst performers of the bunch, for comparison.
I agree the trades only closed because the simulation ended. But I'm not sure I would call it just "luck" as there are exit criteria in the strategy.
Yeah, I saw that about the OptimalF being figured across the enter period. My theory about this (please correct me if I'm wrong) is that I want to throw all the available assets at a strategy and then let OptimalF throw them out if they aren't profitable. If they ARE profitable, then use that OptF factor to weight the trades based on HOW profitable (they have been historically).
I would like to know how to tell if something is wrong here (ie, a testing glitch). The strategy in and of itself seems reasonable, but the results seem overly optimistic. (Then again, I'm not sure what I should expect from a robot.)
Last edited by dusktrader; 07/22/1316:47. Reason: oops wrong pic
Re: Too good to be true? Help me poke holes in it please
[Re: dusktrader]
#426380 07/22/1316:5007/22/1316:50
The OptimalF factors themselves are not the problem, but the way how you use them prevents reversals and thus keeps most trades open indefinitely.
When trades are never closed, the outcome of your system only depends on the random position of the price at the end of the simulation when the exits are enforced. If it's in favorable distance from the average entry, your system makes an extreme win, otherwise an extreme loss. The chance is 50%.
Use a decent exit condition and don't keep your trades open until the end of the simulation.
Re: Too good to be true? Help me poke holes in it please
[Re: dusktrader]
#426384 07/22/1317:0007/22/1317:00
Thanks blaub4r. I'm attaching here also EURUSD's result chart, one of the worst performers of the bunch, for comparison.
I agree the trades only closed because the simulation ended. But I'm not sure I would call it just "luck" as there are exit criteria in the strategy.
If that's true, then it leaves two possibilities:
a) you're working on a strategy that will happily keep trades for years (decades?) and that's fine with you
b) you have a bug in the exit strategy
Originally Posted By: dusktrader
I would like to know how to tell if something is wrong here (ie, a testing glitch). The strategy in and of itself seems reasonable, but the results seem overly optimistic. (Then again, I'm not sure what I should expect from a robot.)
Well, if it looks too good to be true, it probably is. This rule of thumb applies quite nicely to backtesting in general. I too have posted a too optimistic strategy a month ago or so, and it took me hours to find a delicate error that produced too optimistic results. But I was munging with price data, and it was even easier for me to commit a similar mistake. We can't know what you have done wrong unless you post at least the relevant part of the strategy.
But, just as blaub4r, I suspect your exit strategy and think you just got lucky. If it's a reversal strategy, what happens if you just reverse the rules (go long where you now go short, and vice versa)?
Re: Too good to be true? Help me poke holes in it please
[Re: ]
#426385 07/22/1317:3307/22/1317:33
Yet, despite the fact that some trades are kept open forever the equity curve (the blue thing) is incredibly steady. This is very difficult to explain as it does not seem to go down. EDIT: As this rise of the curve starts when the trades which are not closed are entered, you have probably catched some kind of general long-term market movement. I don't know if it is possible to predict or (most likely) just luck. /EDIT If there's no other mistake the exit conditions should not matter that much as you would always be in profit if you close all your trades manually.
However, I got this one when testing your strategy:
forum run.. assets....... Walk-Forward Test: forum portfolio 2009..2013 Read forum.fac forum_1.par forum_2.par forum_3.par forum_4.par forum_5.par forum_6.par forum_7.par Profit 862$ MI 31$ DD 345$ Capital 619$ Trades 966 Win 43% Avg +11.1p Bars 76 AR 94% PF 1.19 SR 0.48 UI 33.3% Error 22% Generate Chart - please wait... ok
I just changed the test period. Now it looks like a normal strategy
Last edited by blaub4r; 07/22/1317:58.
Re: Too good to be true? Help me poke holes in it please
[Re: ]
#426386 07/22/1317:3807/22/1317:38
dusktrader, here's an idea, why not just add TimeExit in your script? That should force the trades to be closed after a defined number of bars. Set it to something you're comfortable with. Then see how it behaves in backtest.
I'm afraid I haven't yet invested enough time in OptimalF factors and Kelly, so won't be able to help with those specific details atm. But I'm boookmarking this thread for the future perusal, because I do have a similar idea to sometime create a strategy that will tactically trade what worked in close past, and skip trades that haven't. What you do here looks very similar to that idea.
Re: Too good to be true? Help me poke holes in it please
[Re: ]
#426393 07/22/1318:3507/22/1318:35
We can't know what you have done wrong unless you post at least the relevant part of the strategy.