Keeping non-24h markets synchronized in live and backtest

Posted By: MattY

Keeping non-24h markets synchronized in live and backtest - 10/27/16 15:03

Hi all,

when I trade a non-24h market, like the FXCM NAS100, I get different live results than in the backtest. That's because Zorro generates bars from the closed market and then the system with its indicators is executed differently than within a backtest (see figures, the red circle shows the surplus bars generated by live Zorro).

I tried to fix this issue with hour() and StartMarket/EndMarket but I get a series error.

How can Zorro be told to just execute the system when the market is open?

/Matt

Attached picture NAS Live.PNG

Posted By: dr_panther

Re: Keeping non-24h markets synchronized in live and backtest - 10/28/16 07:05

You could define the open time of a market like that
(from the manual)

Code:

function run()
{
  ...
  StartWeek = 10400; // start Monday 4 am
  EndWeek = 51900; // end Friday 7 pm
  Weekend = 7; // log off during the weekend
  ...
}

Posted By: MattY

Re: Keeping non-24h markets synchronized in live and backtest - 10/30/16 08:34

Hey, thanks! StartWeek/EndWeek work, but they skip bars only at the beginning and end of the week.I am looking for a function that skips bars once a day.

In the meanwhile I found the TimeFrame function. The manual states that it is for "for skipping bars outside market hours or when no price ticks arrive". I implemented this code to my script.

Code:

// skipping bars for a certain time period 
	static int BarsMissing = 0;
	if(hour() >= 21 and hour() < 22) // 0 when the current bar has no price quotes   
	{ 
  		TimeFrame = 0; // set to zero when not in frame
  		BarsMissing++;
	}
	else if (hour() == 22 and minute() == 0 )
	{ 
  		TimeFrame = -BarsMissing; //  set TimeFrame to the negative number of skipped bars for ending the frame
  		BarsMissing = 0;
	}
	else
		TimeFrame = 1;	// Normal operation

But only the prices and indicator values are fixed for the specified time period and no bars are skipped. Anything wrong with the implementation?

Posted By: jcl

Re: Keeping non-24h markets synchronized in live and backtest - 10/31/16 11:17

I am not sure what you mean with "skipping" bars, but if you want to restrict your trading to market hours, use the hour() or lhour() function for this. You can normally get the market hours for your asset from the broker's website.

Bars are skipped when the asset is not traded and there are no price quotes. This depends on the asset, not on your script.

Posted By: dr_panther

Re: Keeping non-24h markets synchronized in live and backtest - 11/01/16 09:42

I think he wants to just consider particular bar for the calculation of an indicator.

MattY included 2 images, and the indicator is also calculated for some "flat" bars, this bars should be excluded in the calculation of an indicator.

For example lets say I just want to have the EMA from 8:00 - 16:00 UTC, that means every bar between 4pm and 8am is ignored in the calculation, is this possible?

Posted By: MattY

Re: Keeping non-24h markets synchronized in live and backtest - 11/02/16 20:33

Hi dr_panther, jcl,

that is exactly what I mean. I want to skip the bars when the market is closed. For the FXCM NAS100 this is between 20:15 - 20:30 and 21:00 - 22:00 UTC. See image "NAS100 Historical". These times are not in the data history and I also do not receive any live quotes during these times in the FXCM Trading station. But Zorro seems to run its quoting and logging and produces these bars with always the same price. See image "NAS100 Zorro" for an example.

Therefore the Zorro live data is different from the historical data and the indicators are calculated differently. And the whole system behaves differently as in the backtest/simulation. For me this is a big problem.

jcl, is there no way to exclude these times? If not, please consider to implement it in a future version.

Attached picture NAS100 Historical.PNG

Posted By: Finstratech

Re: Keeping non-24h markets synchronized in live and backtest - 11/08/16 22:29

If you want to trade during certain hours try this:

if(between(hour(),15,21)..............)

This will trade from 15:00 to 21:00 UTC

Posted By: MattY

Re: Keeping non-24h markets synchronized in live and backtest - 11/09/16 20:36

Thanks, but if I add an hour() condition to the complete function run() I get this error:

Quote:

Error 041: Inconsistent series calls / too many series
Number or order of series calls in the script are different between run cycles, due to a bug in the script. Make sure that all series calls - or function calls that internally create series, such as LowPass, ATR, etc. - have the same order in every run, and are not skipped with if statements. Make also sure that your script does not generate series in tick-based intrabar functions (TMF, tick) or in endless loops.

And I think it's not enough to put the entry and exit statements in an hour() condition, because I also don't want the indicators to be calculated for a specific time.

/Matt

Posted By: dr_panther

Re: Keeping non-24h markets synchronized in live and backtest - 11/12/16 22:29

I waited for a better solution before making this suggestion:

Would it be possible to remove the unwanted bars from the history file and restart zorro, this should calculate the indicators from the remaining bars and if the deleted bars are not the last ones, it would not reload it from the data provider.

I know its very dirty, but I don't know an elegant way to achieve it, to be honest I even don't know if my suggestion would work.

Posted By: boatman

Re: Keeping non-24h markets synchronized in live and backtest - 11/13/16 00:43

@MattY - you would get that error if you put a series definition inside the if() statement. Series need to be created in the same way and in the same order for each iteration of the run function, therefore you shouldn't put them inside flow control statements.

You could try defining your series outside the if() statement, and then wrapping only your entry logic inside the if() statement. Example:

Code:

vars mySeries = series(...);

if(tradeCondition == true)
{
   Entry = priceHigh();
   enterLong();
}

Posted By: MattY

Re: Keeping non-24h markets synchronized in live and backtest - 11/14/16 20:44

@dr_panther: the history doesn't have unwanted bars. It's the live data that have these additional wrong bars. That's because Zorro's quoting is running when the particular market is closed. And then it produces additional bars that are not in the history.

@boatman: when I put only the entry logic inside the if() statement I get no error that's right. But I can't skip these additional live bars. So, I tried to put the whole run() function inside an if statement to execute the system only during these times.

I am stuck, here. A function like StartWeek/EndWeek would be needed so that Zorro logs out at the end of the day and doesn't produce these surplus bars, something like StartDay/EndDay.

Doesn't anybody else have these live trading problem? It should occur with every non-24h market and on a lot of timeframes.

BTW: I tried it also with IB in demo-mode. It's the same behaviour.

Posted By: Finstratech

Re: Keeping non-24h markets synchronized in live and backtest - 11/14/16 22:39

Play with this:

Code:

// create a midnight-aligned daily price series
static int BarsPerDay = 0;
if(hour(0) < hour(1)) { // day change  
  TimeFrame = -BarsPerDay; // end the frame
  BarsPerDay = 0;
} else {
  TimeFrame = 0; // inside the frame
  BarsPerDay++;  // count number of bars per day
}
vars PriceD1 = series(price());

// alternative: a midnight-aligned price series using frameSync()
StartWeek = 10000;
TimeFrame = frameSync(H24);
vars PriceD1 = series(price());

Also enter "time zone" in the search bar of the HTML Help document located in the main Zorro folder.

Posted By: boatman

Re: Keeping non-24h markets synchronized in live and backtest - 11/15/16 23:56

Finstratech is right - you need to set up the TimeFrame correctly. I typically use AssetZone, and then set TimeFrame to AssetFrame. Then use FrameOffset to sample prices at the time of interest.

Posted By: MattY

Re: Keeping non-24h markets synchronized in live and backtest - 11/18/16 10:41

Hey, I figured it out finally. Actually the code here does the job.

Code:

// skipping bars for a certain time period 
	static int BarsMissing = 0;
	if(hour() >= 19 and hour() < 20) // 0 when the current bar has no price quotes   
	{ 
  		TimeFrame = 0; // set to zero when not in frame
  		BarsMissing++;
	}
	else if (hour() == 20 and minute() == 0 )
	{ 
  		TimeFrame = -BarsMissing; //  set TimeFrame to the negative number of skipped bars for ending the frame
  		BarsMissing = 0;
	}
	else
		TimeFrame = 1;	// Normal operation

So, I had it but I didn't see it. Thanks for pushing me in the right direction, again. The reason is I thought Zorro would produce no log file, when skipping bars. But it does and freezes all indicator and other values when TimeFrame is 0. See here:

Quote:

[2408: Thu 17.11.16 18:55] +0 +205 64/3 4825/4826\4824/4824 Close: 4824.00, Ind: 4824.76, AssetFrame: 1, TimeFrame: 1

[2409: Thu 17.11.16 19:00] +0 +187 64/3 4824/4825\4820/4821 Close: 4824.00, Ind: 4824.76, AssetFrame: 1, TimeFrame: 0

[2410: Thu 17.11.16 19:05] +0 +167 64/3 4820/4821\4816/4817 Close: 4824.00, Ind: 4824.76, AssetFrame: 1, TimeFrame: 0

[2411: Thu 17.11.16 19:10] +0 +167 64/3 4815/4818\4813/4817 Close: 4824.00, Ind: 4824.76, AssetFrame: 1, TimeFrame: 0

[2412: Thu 17.11.16 19:15] +0 +170 64/3 4817/4819\4816/4818 Close: 4824.00, Ind: 4824.76, AssetFrame: 1, TimeFrame: 0

[2413: Thu 17.11.16 19:20] +0 +187 64/3 4818/4821\4818/4821 Close: 4824.00, Ind: 4824.76, AssetFrame: 1, TimeFrame: 0

[2414: Thu 17.11.16 19:25] +0 +167 64/3 4819/4821\4817/4817 Close: 4824.00, Ind: 4824.76, AssetFrame: 1, TimeFrame: 0

[2415: Thu 17.11.16 19:30] +0 +150 64/3 4818/4818\4814/4815 Close: 4824.00, Ind: 4824.76, AssetFrame: 1, TimeFrame: 0

[2416: Thu 17.11.16 19:35] +0 +141 64/3 4814/4815\4811/4813 Close: 4824.00, Ind: 4824.76, AssetFrame: 1, TimeFrame: 0

[2417: Thu 17.11.16 19:40] +0 +160 64/3 4813/4817\4813/4816 Close: 4824.00, Ind: 4824.76, AssetFrame: 1, TimeFrame: 0

[2418: Thu 17.11.16 19:45] +0 +153 64/3 4817/4819\4815/4815 Close: 4824.00, Ind: 4824.76, AssetFrame: 1, TimeFrame: 0

[2419: Thu 17.11.16 19:50] +0 +157 64/3 4815/4818\4814/4816 Close: 4824.00, Ind: 4824.76, AssetFrame: 1, TimeFrame: 0

[2420: Thu 17.11.16 19:55] +0 +149 64/3 4816/4817\4814/4814 Close: 4824.00, Ind: 4824.76, AssetFrame: 1, TimeFrame: 0

[2421: Thu 17.11.16 20:00] +0 +166 64/3 4816/4818\4814/4817 Close: 4817.25, Ind: 4823.35, AssetFrame: 1, TimeFrame: -12

[2422: Thu 17.11.16 20:05] +0 +178 64/3 4818/4820\4817/4819 Close: 4819.40, Ind: 4822.36, AssetFrame: 1, TimeFrame: 1