Mixed frequency history (M1+D)

Posted By: kankan

Mixed frequency history (M1+D) - 05/20/19 09:53

I was wondering whether even in theory is possible to generally have daily history data but only for some of the days have M1 history (and use it).

The motivation is that I trade many assets, but very very infrequently and only intraday for 1 day. So instead of downloading full M1 history for all the assets which would take a month from my broker, I wanted to download just the days I needed for the backtesting. Since those are earnings, they are well defined.

I am a little bit confused if this is possible at all and how I would approach this.
Sounds like AssetFrame and TimeFrame could be useful for skipping periods with no data until the date that has M1 history, then running at 1-minute intervals (or maybe running at 1-minute intervals the whole time and skipping zero-intraday data?)

Also, does the data have to be stored as M1 with many empty periods, or can it be mixed?

Thank you for any pointers.

Posted By: jcl

Re: Mixed frequency history (M1+D) - 05/20/19 10:21

It must be stored as M1 with many gaps. If it's a portfolio, let the first asset be the one with the least gaps, because it determines the bars. If the gaps don't overlap, use a dummy asset as the first.

Posted By: kankan

Re: Mixed frequency history (M1+D) - 05/31/19 13:01

So I assembled the history in UTC, single file per asset, that has a mix of daily and 1-minute bars, with the timestamp at the end of the bar as written in the manual.
I've been reading about it and I am still very confused on how to set up the script to make use of this data. With the below setup, I get "SMA Lookback period exceeded by 286360 bars" error. I am guessing it's expecting 200*1440 minute bars before the StartDate. Do I need to make my own bar() function to make use of this data properly?

The data is stored like this. I don't need empty bars where there is no data, correct?

2018.07.31 13:33 190.02 190.31 189.95 190.21 0 834 // MINUTE
2018.07.31 13:32 190.31 190.34 189.91 190 0 1233 // MINUTE
2018.07.31 13:31 190.34 190.5 190 190.3 0 4165 //MINUTE
2018.07.30 20:00 191.98 192.2 189.07 189.91 0 1.59208E+07 //DAILY
2018.07.27 20:00 194.93 195.19 190.21 190.98 0 1.41448E+07 //DAILY
2018.07.26 20:00 194.61 195.96 193.61 194.21 0 1.43066E+07 //DAILY

The assets file first has a DUMMY asset, then another entry for AAPL

The very basic script where I am opening a long trade on a specific date that has daily resolution data:

Code:

function run() {
	BarPeriod = 1;
	StartDate = 20130301;
	LookBack = 200*1440;  // enough for the 200day SMA
	LifeTime = 448;  // automatically close trades at the end of the day
	assetList("History\AssetsEarnings.csv");
	History = "m.t6";
	asset("");  // not sure if I need the DUMMY here...
	asset("AAPL");
	
	TimeFrame = 1440;  // daily SMA
	vars dailyPrices = series(priceClose());
	vars smaDaily = series(SMA(dailyPrices, 200));
	 
	// buy after open on 2018-07-31... the day that has minute resolution data
	if(year(0)==2018 && month(0)==7 && day(0)==31) { 
		TimeFrame = 1;  // run on minute intervals during trading
		if(ltod(ET, 0)==931 && NumOpenLong == 0) {
			enterLong();
		}
	}
	else {
		TimeFrame = 1440;
	}

Posted By: Petra

Re: Mixed frequency history (M1+D) - 05/31/19 14:15

A 1440 timeframe when you have mostly daily bars is many years long and your lookback period is 1000 years. You must calculate the sma by script with only 1 bar per day, and use a normal lookback period of 200 or so.

Posted By: kankan

Re: Mixed frequency history (M1+D) - 06/01/19 12:42

Thank you Petra. To play with it, I removed the SMA calculation and then things run as expected.

However, I am still confused with what a Bar really refers to (and therefore BarPeriod). I thought it was a time interval, say 1-minute, and regardless of the gaps in the history I have, it will be a fixed number of them between StartDate and EndDate.

But, it seems that it refers to the actual historical bars, more like candles actually. Is this correct? So in my case, a daily candle (when there are no minute candles), is treated as 1 bar, much the same as an intraday 1-minute candle. Therefore, when I set Lookback to 200, that can refer to a varying amount of time actually, depending on the available history before the StartDate?

Am I understanding this correctly?

Also, I am unsure what you mean by "calculate sma by script". Do you mean I manually have to build a series of just daily prices?

Posted By: Petra

Re: Mixed frequency history (M1+D) - 06/02/19 04:48

Yes the bar length depends on historical bars. You really need daily prices for the sma, not 1440 bars.

Posted By: kankan

Re: Mixed frequency history (M1+D) - 06/03/19 16:49

This is how I managed to build a daily SMA, however this seems extremely fragile to me (for backtesting). For example, if there were no trades at 16h ET, there would be no bar and we wouldn't produce a daily close. There must be a more robust way of doing this, no? Is there some way to track an internal clock during backtesting?

Code:

// capture NYSE market close and the Friday close that falls on Sunday 23h UTC
static int numSkippedBars = 0;
if(lhour(ET, 0)==16 || hour(0)==23) {
	TimeFrame = min(-1, -numSkippedBars);
	numSkippedBars = 0;
}
else {
	TimeFrame = 0;
	numSkippedBars++;
}
vars dailyPrices = series(priceClose(0));
vars sma = series(SMA(dailyPrices, 200));

I also tried this, but it doesn't work correctly and I'm not sure why. It correctly gets the daily close when there are intraday bars and on the first day of the week only. Everything else gets skipped (TimeFrame=0). Why?

Code:

FrameOffset = 21*60-dst(ET, 0)*60;
TimeFrame = frameSync(1440);

vars dailyPrices = series(priceClose(0));
vars sma = series(SMA(dailyPrices, 200));

What would be a proper way to build a daily close series?
Thank you.