I'm working on rebuilding dt-e7 (aka "demo5"). I have made a number of changes to my build process and wanted to share. This tradebot is being built under Zorro 1.20 and I expect to trade it live once it's done. I'll share my build process along the way and you can feel free to add improvements or any input really.
The former iterations of this tradebot did not properly handle the Margin=0 situation, instead taking 1 Lot trades. I believe this affected the optimization too much, so I've started over from scratch on this build (note in Zorro v1.20, Margin=0 will no longer open a live trade).
I initially started with EURUSD, but was not very satisfied with the results. I then moved to NZDJPY and was happier with that basis, so I used that asset as the core to build out from. Therefore, in dt-e8 EURUSD is completely unused.
The infrastructure is constantly evolving and (hopefully) improving as well. Almost everything works now, several bugs squashed, and no more known issues as of Zorro 1.20.
I've identified the 7 pairs now that I'll use for this tradebot: USDJPY; EURAUD; EURCAD; AUDUSD; AUDCHF; AUDJPY; NZDJPY
Just for fun, I've calculated the "what if" scenario, if this tradebot were to go live today. But it's only optimized for 1 asset, not the other 6 (that's the work still left to do...) But I can use this scenario as a benchmark for comparison... to see how much better the tradebot will become when its fully optimized.
function fridayClose(int fridayclose)
{
//allows Friday trading up until NYSE 3pm; close trades and don't allow after this
if(fridayclose && dow() == FRIDAY && lhour(ET) >= 15)
{
exitLong("*");
exitShort("*");
return 1; //condition met; indicate no further trades
}
return 0; //condition not met; safe to take new trades
}
function hourOpen(int hourblockstart, int hourblockend)
{
//blocks new open trades between selected hours
//uses NYSE time, including DST
if ( (lhour(ET) >= hourblockstart) && (lhour(ET) < hourblockend) )
return 0; //between blocked hours, do not allow trade opens
else
return 1; //no conditions met, allow trades by default
}
function todayOpenCombo(var dayopencombo)
{
//allows optimizer to specify the best combo of days for opens
//bit position 0 = Monday
//bit position 1 = Tuesday
//bit position 2 = Wednesday
//bit position 3 = Thursday
//bit position 4 = Friday
//bit position 5 = Sunday
//given a combination #, the function will return whether
//current dow() is in the combination
int dayopencombobits = dayopencombo+.5; //truncate to rounded int
int today = dow() - 1; //Mon is 0
if (today == 6) today = 5; //bump Sun to 5 (no Sat, keep binary range 0-63)
if (dayopencombobits & (1 << today)) return 1; //current dow() is in the combo
else return 0; //current dow() not in combo, do not allow trade opens
}
function todayCloseCombo(var dayclosecombo)
{
//allows optimizer to specify the best combo of days to close by NYSE 4pm
//bit position 0 = Monday
//bit position 1 = Tuesday
//bit position 2 = Wednesday
//bit position 3 = Thursday
//bit position 4 = Friday
//bit position 5 = Sunday
//given a combination #, the function will determine if we are beyond
//a combo close time, close all trades if necessary, and return 1
//if no further trades allowed today
int dayclosecombobits = dayclosecombo+.5; //truncate to rounded int
int today = dow() - 1; //Mon is 0
if (today == 6) today = 5; //bump Sun to 5 (no Sat, keep binary range 0-63)
if ((dayclosecombobits & (1 << today)) && lhour(ET) >= 16)
{
exitLong("*");
exitShort("*");
return 1; //current dow() is in the combo; indicate no further trades
}
else return 0; //current dow() not in combo, safe to take new trades
}
function marketOpenCombo(var marketopencombo)
{
//allows optimizer to specify best markets to initiate trades
//bit position 0 = New York 8am-5pm Eastern
//bit position 1 = Sydney 5pm-2am Eastern
//bit position 2 = Tokyo 7pm-4am Eastern
//bit position 3 = London 3am-12pm Eastern
//given a combination #, the function will determine if current time is within
//a market part of the combination (returns 1 to allow trading if true)
int marketcombobits = marketopencombo+.5; //truncate to rounded int
if ( (lhour(ET) >=8) && (lhour(ET) <17) && (marketcombobits & (1 << 0)) ) return 1; //inside New York
if ( (lhour(ET) >=17) || (lhour(ET) <2) && (marketcombobits & (1 << 1)) ) return 1; //inside Sydney
if ( (lhour(ET) >=19) || (lhour(ET) <4) && (marketcombobits & (1 << 2)) ) return 1; //inside Tokyo
if ( (lhour(ET) >=3) && (lhour(ET) <12) && (marketcombobits & (1 << 3)) ) return 1; //inside London
return 0; //default - current market not in combination, don't allow trade opens
}
function checkTradesPerCycle()
{
//require minimum 30 trades per WFO cycle or stop training
static int LastWFOCycle = 0, LastNumTrades = 0;
if(Train && (WFOCycle != LastWFOCycle) )
{
if(LastNumTrades > 0 and LastNumTrades < 30)
{
char tradecount[100];
sprintf(tradecount,"Not enough trades per cycle: %d",LastNumTrades);
quit(tradecount);
}
LastWFOCycle = WFOCycle;
}
LastNumTrades = NumWinTotal+NumLossTotal;
}
function calculateMargin(int direction)
{
//calculate risk Margin based on OptimalF and trade direction
Capital = 1000; //simulated account balance
var riskCapital = 300; //basis to trade with
if (direction && OptimalF>.001) //long trade, historically profitable
{
Lots = 1; //allow live trading
return OptimalFLong * riskCapital;
}
else if (!direction && OptimalF>.001) //short trade, historically profitable
{
Lots = 1; //allow live trading
return OptimalFShort * riskCapital;
}
else
Lots = -1; //non-historically profitable = phantom live trades only
return 0; //no Margin allocated for non-historically profitable
}
function getOpt(string param)
{
switch (Asset)
{
case "EURUSD":
switch (param)
{
case "timecycle": return optimize(75,65,85,1,0);
case "timefactor": return optimize(2.4,0.2,5,0.2,0);
case "stop": return optimize(7.5,2,15,0.5,-1);
case "trail": return optimize(8,3,11,0.5);
case "maxtrades": return 5; //limit trades per signal
case "marketopencombo": return optimize(13,1,15,1); //15)every market
case "dayopencombo": return 63; //optimize(61,1,63,1); //63)every day
case "dayclosecombo": return 0; //optimize(33,1,63,1); //0)none; 63)every day
case "fridayclose": return 1; //no trades after NYSE 3pm Friday
case "reversedir": return 0; //trade opposite signals: 0)normal trade direction
case "hedge": return 0; //Hedge mode: 0)none, 4)virtual
case "hourblockstart": return 0; //block trade opens beginning at NY hour
case "hourblockend": return 0; //block trade opens ending at NY hour
case "lossstreaks": return 21; //0)no limit
case "emode": return 0; //equity-curve trading: 0)none; 1)standard; 2)switch hitter; 3)weighted; 4)mean reversion
}
case "NZDJPY":
switch (param)
{
case "timecycle": return optimize(70,60,80,1,0);
case "timefactor": return optimize(2.4,0.2,5,0.2,0);
case "stop": return optimize(4,1,7,0.5,-2);
case "trail": return optimize(7,3,11,0.5);
case "maxtrades": return 7; //limit trades per signal
case "marketopencombo": return 15; //optimize(9,1,15,1); //15)every market
case "dayopencombo": return 63; //optimize(13,1,63,1); //63)every day
case "dayclosecombo": return 0; //optimize(34,1,63,1); //0)none; 63)every day
case "fridayclose": return 0; //no trades after NYSE 3pm Friday
case "reversedir": return 0; //trade opposite signals: 0)normal trade direction
case "hedge": return 4; //Hedge mode: 0)none, 4)virtual
case "hourblockstart": return 0; //block trade opens beginning at NY hour
case "hourblockend": return 0; //block trade opens ending at NY hour
case "lossstreaks": return 19; //0)no limit
case "emode": return 1; //equity-curve trading: 0)none; 1)standard; 2)switch hitter; 3)weighted; 4)mean reversion
}
default: //use with Step 3a: prospecting
switch (param)
{
case "timecycle": return optimize(70,60,80,1,0);
case "timefactor": return optimize(2.4,0.2,5,0.2,0);
case "stop": return optimize(4,1,7,0.5,-2);
case "trail": return optimize(7,3,11,0.5);
case "maxtrades": return 7; //limit trades per signal
case "marketopencombo": return 15; //optimize(9,1,15,1); //15)every market
case "dayopencombo": return 63; //optimize(13,1,63,1); //63)every day
case "dayclosecombo": return 0; //optimize(34,1,63,1); //0)none; 63)every day
case "fridayclose": return 0; //no trades after NYSE 3pm Friday
case "reversedir": return 0; //trade opposite signals: 0)normal trade direction
case "hedge": return 0; //Hedge mode: 0)none, 4)virtual
case "hourblockstart": return 0; //block trade opens beginning at NY hour
case "hourblockend": return 0; //block trade opens ending at NY hour
case "lossstreaks": return 0; //0)no limit
case "emode": return 0; //equity-curve trading: 0)none; 1)standard; 2)switch hitter; 3)weighted; 4)mean reversion
}
}
}
function checkEquity()
{
int emode = getOpt("emode");
if (!emode) return; //no equity-curve trading
//emode 1 = standard: sets phantom/normal mode only (via Lots)
//emode 2 = switch hitter: always in market (Lots=1), fades direction (via dir)
//emode 3 = reward success with weighting: increase trades based on degree of improvement
//emode 4 = mean reversion: trade when equity curve falls (Lots=1), sit out when it rises (Lots=-1)
vars EquityCurve = series(EquityLong+EquityShort); //includes all phantom equity
var dir; //indicates normal trade direction (dir=1) or reverse (dir=-1)
//narrower curves
//var slow = 50;
//var fast = 10;
//wider curves
//var slow = 100;
//var fast = 10;
//mega-wide curves
var slow = 200;
var fast = 10;
//uber-wide curves
//var slow = 300;
//var fast = 10;
//optimized curves
//var slow = optimize(50,50,300,12);
//var fast = 10;
vars EquityLP = series(LowPass(EquityCurve,fast));
var EquityLPfalling = LowPass(EquityLP,slow);
var EquityLPrisingBigger = LowPass(EquityLP,slow*3.2);
var EquityLPrisingBig = LowPass(EquityLP,slow*1.5);
//plot("EquityLPslow",LowPass(EquityLP,slow),1,BLUE);
//plot("EquityLPfast",LowPass(EquityLP,fast),0,GREEN);
if(EquityLP[0] < EquityLPfalling && falling(EquityLP)) { //drawdown
if (emode==1) Lots = -1; //set phantom trade mode
if (emode==2) return 1; //fade: take signals when losing
if (emode==3) { //reward success with weighting
Lots = -1; //set phantom trade mode
return 1; //allow max 1 phantom trade in drawdown
}
if (emode==4) Lots = 1; //mean-reversion: start trading when equity curve falls
}
else { //positive equity curve
if (emode==1) Lots = 1; //set normal trade mode
if (emode==2) return -1; //fade: take reverse signals when winning
if (emode==3) { //reward success with weighting
Lots = 1; //set normal trade mode
if (EquityLP[0] > EquityLPrisingBigger && rising(EquityLP)) return 3; //very big rising
else if (EquityLP[0] > EquityLPrisingBig && rising(EquityLP)) return 2; //big rising
else return 1; //rising but not yet significantly
}
if (emode==4) Lots = -1; //mean-reversion: stop trading when equity curve rises
}
}
function checkStreak()
{
//disallow trading this asset if loss streak threshold breached
if (getOpt("lossstreaks") && ((LossStreakLong>=getOpt("lossstreaks")) || (LossStreakShort>=getOpt("lossstreaks"))))
{
Margin = 0;
printf("\nWarning: %s breached loss streak limit of %i; may need reTrain",Asset,getOpt("lossstreaks"));
}
}
function checkModifiers()
{
int reversedir = getOpt("reversedir"); //default 0=normal trade direction
int fridayclose = getOpt("fridayclose"); //enforce auto-close and no trades after NYSE 3pm Friday
int hourblockstart = getOpt("hourblockstart"); //block trade opens beginning at NY hour
int hourblockend = getOpt("hourblockend"); //block trade opens ending at NY hour
int dayopencombo = getOpt("dayopencombo"); //combo of days to open; 63=every day
int dayclosecombo = getOpt("dayclosecombo"); //combo of days to close after NYSE 4pm; 0=none; 63=every day
int marketopencombo = getOpt("marketopencombo"); //combo of markets to allow trade opens; 15=every market
if ( (!fridayClose(fridayclose) //close NYSE 3pm on Friday
|| !todayCloseCombo(dayclosecombo) ) //close NYSE 4pm on selected days
&& todayOpenCombo(dayopencombo) //open on selected days only
&& marketOpenCombo(marketopencombo) //open during selected markets only
&& hourOpen(hourblockstart,hourblockend) ) //open during selected hours only
return 1; //ok to place new trades
else
return 0; //no trade, restricted by a modifier
}
function run()
{
AssetList = "Assets-IBFXnano.dta";
//AssetList = "Assets-IBFXnano-spread100.dta";
//AssetList = "Assets-IBFXnano-spread300.dta";
//AssetList = "Assets-FXCMmicro.dta";
//AssetList = "Assets-FXCMmicro-spread100.dta";
set(PARAMETERS+ALLCYCLES+FACTORS);
StartDate = 20080101;
EndDate = 20131125;
GapDays = 3; //alert if more than 3 days gap in data
BarPeriod = 15;
LookBack = 600;
DataSplit = 70; //70% training, 30% OOS test
NumWFOCycles = 5;
if(is(TESTMODE)) NumSampleCycles = 15; //oversampling on Test only, not Train
if (Train) { RollLong = 0; RollShort = 0; } //help prevent asymmetry in parameters & profit factors
checkTradesPerCycle(); //stop Train early if not enough trades
int reinvestprofits = 1; //invoke margin setting during trade logic
while(asset(loop("USDJPY","EURAUD","EURCAD","AUDUSD","AUDCHF","AUDJPY","NZDJPY")))
//asset("NZDJPY");
{
int maxtrades = getOpt("maxtrades");
checkEquity(); //equity-curve trading
Hedge = getOpt("hedge");
//edge trading logic
var TimeCycle = getOpt("timecycle");
var TimeFactor = getOpt("timefactor");
//Stop = BarPeriod*PIP; //simple stop level
Stop = ATR(100) * getOpt("stop");
Trail = ATR(100) * getOpt("trail");
vars Price = series(price(0));
vars MA1 = series(SMA(Price,TimeCycle));
vars MA2 = series(SMA(Price,TimeCycle*TimeFactor));
if (checkModifiers()) //only evaluate signals if no restrictions in force
{
if (crossOver(MA1,MA2) && rising(MA1)) //long signal
{
if (reinvestprofits) Margin = calculateMargin(1); //long
checkStreak(); //disallow trading if loss streak limit breached
//enterLong(); //standard entry
if (!getOpt("reversedir")) reverseLong(maxtrades); else reverseShort(maxtrades);
}
else if(crossUnder(MA1,MA2) && falling(MA2)) //short signal
{
if (reinvestprofits) Margin = calculateMargin(0); //short
checkStreak(); //disallow trading if loss streak limit breached
//enterShort(); //standard entry
if (!getOpt("reversedir")) reverseShort(maxtrades); else reverseLong(maxtrades);
}
}
}
PlotWidth = 1100;
PlotHeight1 = 600;
}
Re: dt-e8 MA-crossover, built for Zorro v1.20
[Re: dusktrader]
#433365 11/28/1315:5711/28/1315:57
Please feel free to ask any questions about any parts you don't understand. I am actually a total n00b myself, I've just been working on this nonstop. I'm always looking for ways to make it easier to understand. I've found that even very slight changes (or mistakes) can cause dramatic swings in how the tradebot performs. For that reason, I try to have every process well thought out. If you look at the spreadsheet process, you'll see I have a very specific order of tasks and requirements. There is usually a reason for each one, so please do ask if you are trying to follow and have any question.
Btw I think I started on Zorro 1.12, so I haven't been around that long!
Re: dt-e8 MA-crossover, built for Zorro v1.20
[Re: dusktrader]
#433764 12/06/1314:5212/06/1314:52
I'm really happy with this now, and I'm doing the final Train for my demo account. This bot is "done" now, at least in its current form.
I've made a couple enhancements since the first post in this thread: The bot now uses very fresh data, up until 11/23/2013. Also, I have started using the OPENEND flag.
After fully optimizing the 7 pairs, I noticed that some pairs were given a very low OptimalF fraction from looking at the .fac file:
So as an experiment, I told Zorro not to trade anything that produced an OptF less than .1 (originally I had used .001 as the threshold). I was very happy with these results, as you can see in the comparison metrics graphic. It shows that by requiring a minimum 0.1 OptF to trade, one pair was dropped. That only affected the return in a negligible way, but reduced trades by 26% which is a huge reduction in risk for very small opportunity cost.
There are many other ideas I would like to implement, but I'm ready to throw this bot on a demo account and let it play. The next bot I'll design will become dt-e9 (evolution 9).
Aside from the fact that I'd like to work on a new core logic (I'm thinking to experiment with IBS, which is listed in another thread here somewhere)... I do also have several ideas I'm working on for the infrastructure: * Equity-curve trading works now, but it could be better. I will plan to implement some improvements here; * Available margin check: I am always looks for features like this, which will help constrain the tradebot to its statistical simulation bounds. Though the bot performs well in simulation, I would like to ensure that it automatically restricts itself in the event that live trading does not behave as expected (global thermonuclear war??) * File-based state information: I would like to implement a feature to easily save certain state information in a file. While this is currently being done on some level with the .trd files, that is a bit foreign to me and I don't know if we can even access that information. The info I want to save would include metrics such as loss streaks, that could then be used as ongoing constraints even if Zorro were stopped/started multiple times (which is realistic) * I'd also like to play more with the prospecting process. I developed this during dt-e8, so it is definitely in its infancy. One thing I learned is that, despite my efforts to pick a good starting point to prospect from (NZDJPY)... it seems my pick was only mediocre at best (at least according to OptimalF). I've got some ideas I'd like to try on the next bot. I think the danger in this will always be avoiding too much selection bias.
Re: dt-e8 MA-crossover, built for Zorro v1.20
[Re: dusktrader]
#434741 12/25/1310:1512/25/1310:15
I was trying to go over the code for the modifiers yesterday. Most of it made clear sense to me. But there were some conditions, mainly the bitwise ones that I have no idea how they work.
Code:
function todayCloseCombo(var dayclosecombo)
{
//allows optimizer to specify the best combo of days to close by NYSE 4pm
//bit position 0 = Monday
//bit position 1 = Tuesday
//bit position 2 = Wednesday
//bit position 3 = Thursday
//bit position 4 = Friday
//bit position 5 = Sunday
//given a combination #, the function will determine if we are beyond
//a combo close time, close all trades if necessary, and return 1
//if no further trades allowed today
int dayclosecombobits = dayclosecombo+.5; //truncate to rounded int
int today = dow() - 1; //Mon is 0
if (today == 6) today = 5; //bump Sun to 5 (no Sat, keep binary range 0-63)
if ((dayclosecombobits & (1 << today)) && lhour(ET) >= 16)
{
exitLong("*");
exitShort("*");
return 1; //current dow() is in the combo; indicate no further trades
}
else return 0; //current dow() not in combo, safe to take new trades
}
The above code will be a great example if you can elaborate.
Code:
int dayclosecombobits = dayclosecombo+.5;
and this
Code:
if ((dayclosecombobits & (1 << today)) && lhour(ET) >= 16)
Re: dt-e8 MA-crossover, built for Zorro v1.20
[Re: ]
#435750 01/13/1410:5801/13/1410:58
Hi liftoff, I'm very sorry I somehow missed your post in this thread. Must've been too much eggnog!
Anyway, feel free to reply back if you still have more questions about this.
Bitwise operators are not something I'm an expert at, but I spent some time studying them and decided they could be really useful in certain situations, such as finding optimal combinations. I think of them as a panel of dip switches... they can each be on or off, and there are many different combinations.
Code:
int dayclosecombobits = dayclosecombo+.5;
I need the value to be an integer and I also wanted it rounded. Sometimes optimizer returns non-whole numbers so I always want the closest integer. For example it could return 7.98 and in that case I would want to use integer 8. If you add .5 to any var, and then truncate the entire decimal portion, you always end up with the rounded integer (assuming you round from .5 or higher, as I do). Someone else on the forum pointed out that if you set an int to a var value, that Zorro will just truncate the decimal portion -- that saves a step of having to use a round function.
Code:
if ((dayclosecombobits & (1 << today)) && lhour(ET) >= 16)
The if operation is checking to see if the bit position assigned to "today" is currently set in dayclosecombobits. If it is, and if the current (Eastern) time is >= 16:00 hours, then the condition is true.
Re: dt-e8 MA-crossover, built for Zorro v1.20
[Re: dusktrader]
#435763 01/13/1416:4401/13/1416:44
Dusk, First off thank you for donating your code to the community! Awesome stuff and I'm learning a lot from dissecting it (And that is a KILLER spreadsheet!). Small problem, the code posted does not compile (stuck at dt-e8 compiling....). I have all the history data loaded so I'm not sure what could be the cause.
Last edited by Mandark; 02/16/1402:18.
Re: dt-e8 MA-crossover, built for Zorro v1.20
[Re: Mandark]
#437424 02/16/1417:0602/16/1417:06
