Multiple timeframes: prevent loading unnecessary data

Posted By: NorbertSz

Multiple timeframes: prevent loading unnecessary data - 01/22/22 14:47

Hello!

I want to use my script for analyzing multiple timeframes, 5 minutes to daily, all with 200 bars LookBack.
As I understand that the normal way to do that is something like this:

Code

BarPeriod = 5;
LookBack = 12*24*200;             // = 5 minutes * 12 * 24    (1 day)    *   200

TimeFrame = 1;
//...analyze the last 200 bars of M5...
TimeFrame = 3;
//...analyze the last 200 bars of M15...
TimeFrame = 12;
//...analyze the last 200 bars of H1...
TimeFrame = 12*4;
//..analyze the last 200 bars of H4...
TimeFrame = 12*24;
//..analyze the last 200 bars of D1...

My problem with this method is the unneccessarly huge amount of data loaded to RAM. Because with live trading (MT4 brigde) the script loads 12*24*200 bars of 5 minutes data per Asset to be able to calculate the Daily bars. But I just only need the last 200 bars for all of the TimeFrames, so basically 12*23*200 bars are useless at 5 minutes.

For larges timeframes, it could be much better and faster loading directly the larger bars, not calculate it by the smallers. Therefore I don't need to load 12*23*200 bars per asset, only 5*200.

Is there a way to do that?

Thank you!

Posted By: Zheka

Re: Multiple timeframes: prevent loading unnecessary data - 01/22/22 17:12

You can set LookBack =200, if you indeed need only 200 data points of each timeframe.

If it varies - try restricting length of series as needed : vars cls1H = series (priceC(), 200);

Posted By: NorbertSz

Re: Multiple timeframes: prevent loading unnecessary data - 01/22/22 21:09

When I do this in run():

Code

LookBack = 200;
StartDate = Now;
asset("EURUSD");
BarPeriod = 1;
TimeFrame = 1;
vars Price = series(priceOpen());

I see this in Zorro terminal:

Code

...
Load EURUSD prices.. 600 min, gap 1378 min
Trade: testBarperiod EURUSD 2022-01-22
Lookback 200 bars, 2022-01-21..2022-01-21.
...

When I do this:

Code

LookBack = 200;
StartDate = Now;
asset("EURUSD");
BarPeriod = 1;
TimeFrame = 1000;   //!!!!
vars Price = series(priceOpen());

I see exactly the same in Zorro terminal:

Code

...
Load EURUSD prices.. 600 min, gap 1378 min
Trade: testBarperiod EURUSD 2022-01-22
Lookback 200 bars, 2022-01-21..2022-01-21.
...

It's impossible that 600 mins are enough since these are 1000 minutes bars, and I need 200 of them for LookBack. So LookBack is calculated only by BarPeriod, regardless of TimeFrame.

That's why I should do the LookBack in this case for proper operation:

Code

BarPeriod = 1;
LookBack = 1*1000*200 // It's just 200 when I switch to TimeFrame=1000;

...what I want to avoid because of startspeed and memory savings.
According to the code above, for my calulcations I only need:

- 200 bars of 1-minute
- 200 bars of 1000-minute

but this way I get 200,000 pieces of 1-minute, and calculate dynamically the 200 bars of 1000-minutes.

Posted By: Zheka

Re: Multiple timeframes: prevent loading unnecessary data - 01/22/22 22:06

1) to create 200 of 1000min candles you need/strongly advised to set(PRELOAD) such history from a *.t6 file.
2) then , try my second suggestion (and read the manual on series() )

Posted By: Petra

Re: Multiple timeframes: prevent loading unnecessary data - 01/23/22 08:41

Limit the series to 200, because that saves memory. But you will always need 12*24*200 lookback bars for that strategy. You cannot change the bar period during the lookback. Only the time frame.

Posted By: NorbertSz

Re: Multiple timeframes: prevent loading unnecessary data - 01/23/22 10:19

Okay, I see, thank you for the answers!
But there is two problems with this method.

As testing I choosed M1 to H4 timeframes, and calculations for 400 bars:

Code

BarPeriod = 1;
LookBack = 1 * 60 * 4 * 400;

Problem 01

I attached a plotted result. As you can see there are a lot of useless data here, because my broker do not even store this amount of M1 candles. But I need them for calculating the old H4 candles. These are easily avaliable of couse if I go by BarPeriod = 60*4, TimeFrame = 1. But the above way I can't get the old H4 bars, because these are calculated by Zorro with the M1 candles - wich are not exists.

(I also want to avoid that I mix the previously downloaded historical M1 data and my broker's data.)

Problem 02

First loading of months of M1 data is incredibly slow and unneccessary if I calculating with 60 assets.

Possible solution

A solution can be that I run multiple Zorro instances - each one with different BarPeriod, and not changing TimeFrame at all. These scrips could be fast and simple, but for this I need to make a messaging bridge between them - for example some TXT files or interprocess communication.

Have you got a better idea?
Maybe I miss something? Maybe there is a way to download H4 candle data even if my BarPeriod is 1?

Attached picture m1.png

Posted By: Petra

Re: Multiple timeframes: prevent loading unnecessary data - 01/23/22 11:59

Use M1 data and PRELOAD. As Zheka said.

https://manual.zorro-project.com/mode.htm

Posted By: NorbertSz

Re: Multiple timeframes: prevent loading unnecessary data - 01/24/22 15:52

Thank you!

Posted By: NorbertSz

Re: Multiple timeframes: prevent loading unnecessary data - 07/29/22 18:18

I'm stuck with my strategy, because the solutions mentioned above are not meeting the demands.

My problem is still that I need exactly X bars of H4 data and exactly X bars of M1 data. Unfortunately I can't PRELOAD the whole amount of these data from the computer, because:

Huge amount of RAM, unnecessarily, because Zorro's way is loading X*60*4 M1 to generate the needed X H4 bars. So it means (X*60*4 - X) amount of unnecessary M1 data: I only need X of that.
I want flexible code to run on multiple assets, anytime, just by pressing the Trade button. I don't want to start hunting for historical .t6 data for every time when I start trading, with 60+ assets I trade, especially for exotic ones. And theoretically it's not a problem, because my broker gives me the bars I need for my strategy. But I just don't know how can Zorro handle in that way, so that's why it's a problem.
Plus, maybe the historical prices that I can get in .t6 to PRELOAD from an independent source has maybe not the same prices that my broker shows me in the trading session after the LookBack. So I need the data exactly from my broker, not from outer sources.

So if I would make a single-timeframe strategy, it woud be very-very easy to get these X bars of datas: my broker just serves it by BarPeriod = 1, or BarPeriod = 60*4. The problem comes in when I want to use them together, in a multi-timeframe strategy, because Zorro forces to load X*60*4 M1 data to generate the X amount of H4 data - but this amount of M1 data is not avaliable from my broker - and also the RAM problem as mentioned above.

I only want to load X amount of M1 candles and X amount of H4 candles, AND for higher bars: not calculating, but loading from the broker.
For example it would be perfect if Zorro would load the datas 2 times at start: X bars for M1, and X bars for H4.
Is it possible somehow? Do you know an other solution?

In live trading I could run two instances of Zorro, one is in BarPeriod = 1, and the other is in BarPeriod = 60*4, then make some communications between them. But of course I want to backtest the whole strategy, and I would like to not change the code for testing.

I appreciate any ideas about reaching these goals.

Thank you!

Posted By: Petra

Re: Multiple timeframes: prevent loading unnecessary data - 07/30/22 15:09

If you want X bars M1 data and X bars H4 data, but want to avoid generating the H4 from M1, read them from two different datasets. One for H4 and one for M1. Or get H4 from a pre-loaded dataset and load the M1 data as lookback period from the broker.

I think using 2 Zorros for this is guns on sparrows.

Posted By: NorbertSz

Re: Multiple timeframes: prevent loading unnecessary data - 07/30/22 17:14

Using different datasets to the different timeframes would be exactly the solution I am looking for. Could you give me a clue how can I achieve this?

If I understand you well, I need to do the following, by two different scripts (because of the different BarPeriod)

Code

[loader.c]
- start a loader script, BarPeriod = 60*4, LookBack = X
- load the X H4 bar data by iterating every assets, from the broker
- save as file, as a CSV dataset
[trader.c]
- start the trading script: BarPeriod = 1, LookBack = X
- load the H4 dataset from file, and use the prices to analyze
- analyze M1 bars normally, make the decisions, make trading

And everytime I start trading, I need to first run the loader to download the H4 bars as datasets.
Am I correct?

I see the dataset help page,
https://zorro-project.com/manual/en/data.htm
but still can't figure out how can I use datasets it in the same way as backesting as realtime trading, because I need to update somehow the H4 dataset time-by-time, during the backtesting and also in the trading

Posted By: Petra

Re: Multiple timeframes: prevent loading unnecessary data - 07/31/22 08:16

Suppose your H4 data is in column 1 of dataset H, then

var PriceH4 = dataVar(H,dataFind(H,wdate(0)),1);

Posted By: NorbertSz

Re: Multiple timeframes: prevent loading unnecessary data - 08/01/22 13:50

So a possible solution:

When I start the "trader.c" script (BarPeriod = 1), in it's INITRUN it executes another script, let's call it "loader.c":
The "loader.c" (BarPeriod = 60*4) does nothing else but downloads/saves the H4 bars to datasets. It should read a parameter somehow from the "trader.c", because it should work the same way as I run the "trader.c" as Test mode (via an old StartDate), as with Trade mode. So it reads the mode and the StartDate, it goes through the LOOKBACK period, then saves the datas of assets, then imediately just quit(), giving back the processor to the "trader.c" script.
Then the "trader.c" script loads the assets' H4 datas to vars, then uses it to the Testing or the Trading session. And of course, the whole process mentioned above is just for start the testing/trading - after start, the "trader.c" script should take care of updating the H4 vars in the future, everytime a new H4 bar arrives.

And this is the only way I can use exactly the X amount of H4, and X amount of M1, with avoiding the unnecessary bar loadings.

Petra, did you mean this process, or am I overcomplicating the whole thing?

Posted By: Petra

Re: Multiple timeframes: prevent loading unnecessary data - 08/02/22 10:32

Yes it sounds complicated. I would not use 2 scripts but only one. Load the data at start with assetHistory in a dataset. You'll get H1, not H4, but you can then simply use only every 4th record.