bar aggregation problem + Download from FXCM problem

problem #1 - bad bar aggregation
- Zorro 1.66
- downloaded the EURUSD data from FXCM using Zorro's Download script.
- aggregated to Period = 60
- bars are close but not the same as FXCM bars. Clearly not the same
- see picture.
- surprising bad bug for an algo-trading software
- suspect: 1min offset bug due to Zorro timestamp the end of the bar while FXCM timestamps the start of the bar -- only a suspect
- installed Zorro 1.74
- could not re-test because...
problem #2
the Download from FXCM only worked once
- the login isnt successful - red light.
- FXCM demo account works with Trading Station.

Hi,
the bar aggregation bug ("1-min offset")has been fixed in v1.74.6-8.
Another reason for the difference might be Zorro's using Ask prices.

Hello

I installed Zorro 1.74 - and reloaded FXCM data

the problem is still there: Zorro uses the first 1Min bar of the next hour to close the current hour.

Basically the 1min offset problem. see attched

I looked at IB's data and can confirm that there is a problem.

It is then with Zorro/FXCM plug-in incorrect timestamping of data (should be end-of-bar).

There has been recently a report of such a problem, but it could not be reproduced - bars are fine. Still, if you find a wrong bar, just contact Support with the log and the details and they will check it.

You mean that FXCM timestamps are the start of the bar but Zorro thinks it's the end of the bar?

JCL,
I just checked it with a generic MT4 plug-in: downloaded M1 bars have "start-of-the-bar" time stamps.

Yes.
This has been fixed for IB a month or so ago; but now turns out this also is a problem for FXCM and MT4.

It seems that MT4's timestamps are bar open time
https://book.mql4.com/functions/datetime

They probably are, but an MT4 plug-in/Zorro should be storing the data uniformly regardless of a source, sticking to its stated convention.

thanks for your reply.
just open these 2 charts and look at the bars numbered the same.
they are clearly different.
the red/green one is TradingView FXCM feed. and is identical (as it should be) to FXCM Trading Station.
It is quite different from Zorro.

What I suspect is that this is caused by Zorro reading the timestamp of a bar as the End of the Bar, while FXCM (as most) use the timestamp indicating the start of the bar.

You can understand what happens here.... the first 1-min bar of the hour gets assigned as the last 1min bar of the previous hour.

I am really really surprised that no-one has ever noticed it.

Imagine how many back test were conducted with 'wrong' bars...

Luisd, please contact support: support@opgroup.de

This is a nasty bug (

M1 timestamps of brokers are often the bar start. In that case the plugin adds the bar duration to the timestamp. So whatever the problem is, it is probably not the bar start or end time. Any chart looks different anyway, dependent on bid/ask or local/server time. Best contact Support and send them the data.

listen chief engineer

what if I told you that the M1 candles looks exactly the same while the M5 candles does not even look close?

can you draw your conclusion?

An algo trading platform is not a game and messing up with the bars is a very nasty bug.

If M1 candles are identical, but M5 candles are not, then it is no bar start/end problem. It can be simply related to the charting software. It is also known that FXCM historical timestamps have been off several times. But it makes no sense to speculate - just contact Support with your data and they'll look into it and tell you if there is a bug or not. I cannot tell it from here.

JCL,

IT IS NOT ONLY ABOUT FXCM!

I reported exactly this "wrong bar timestamping" problem for IB and it had been fixed. "Bar aggregation problem" has been ignored-publicly- but fixed silently.

Now, it turns out that FXCM AND MT4 plugins have exactly the same timestamping problem (with historical data).

And bar aggregation to 5-min on the just loaded data from an MT4 broker looks like this:
(v 1.78.5)
[6: Fri 18-02-02 08:04] (0.73706)
[7: Fri 18-02-02 08:05] (0.73707)
[8: Fri 18-02-02 08:06] (0.73707)
[9: Fri 18-02-02 08:07] (0.73708)
[10: Fri 18-02-02 08:08] (0.73687)n ETtime=308, 0.73694 0.73724 0.73686 0.73687

[11: Fri 18-02-02 08:09] (0.73671)
[12: Fri 18-02-02 08:10] (0.73643)
[13: Fri 18-02-02 08:11] (0.73644)
[14: Fri 18-02-02 08:12] (0.73636)
[15: Fri 18-02-02 08:13] (0.73634)n ETtime=313, 0.73687 0.73688 0.73627 0.73634

Or - on AUD/USD data supplied with Zorro

[2861: Tue 17-05-09 22:47] (0.73499)
[2862: Tue 17-05-09 22:48] (0.73503)
[2863: Tue 17-05-09 22:49] (0.73504)
[2864: Tue 17-05-09 22:50] (0.73497)
[2865: Tue 17-05-09 22:51] (0.73515)n ETtime=1851, 0.73517 0.73517 0.73492 0.73515

[2866: Tue 17-05-09 22:52] (0.73508)
[2867: Tue 17-05-09 22:53] (0.73508)
[2868: Tue 17-05-09 22:54] (0.73496)
[2869: Tue 17-05-09 22:55] (0.73500)
[2870: Tue 17-05-09 22:56] (0.73490)n ETtime=1856, 0.73515 0.73519 0.73489 0.73490


But the same script (TimeFrame=5; if (frame(0)) printf(..) works as expected on data from IB!!

I also find it weird that Support refuses to read the forum.
At least, this has been my experience.

One has to waste time re-packaging the message,etc instead of just sending a link to the thread. And messages to you on the forum are normally well laid and problems properly documented.

Bug reports on a forum are fine when I happen to see them and when the problem is obvious. If not, the user should contact support. FXCM timestamps are corrected and issues with 5 minutes bars could not be reproduced, so this problem is not obvious. We just need the data that the user anyway has, then we can determine in 5 minutes whether it is a bug or not. No charts, paintings and lengthy discussions are needed. I can really not see what the problem is.

The general problem is that the "burden of proof" is put fully on a user and the default attitude is "we are great and fine until proven otherwise".
And it takes maaany messages to move the needle.

I spend on average 20-30min to properly word a message, format it and support it with code and log.
The answer, however, is often not fully addressing the problem and requires another 1-3 messages pressing from different angles to get to a specific next step.
Good recent example (not mine) is the discussion about "weekend" behaviour of FXCM AND IG plug-ins (or Zorro)..

Similarly, several messages above I wrote that I had verified that MT4 plug-in was not timestamping M1 bars properly. That was just ignored.
Is my last message with several log details not enough to see that there most probably is a problem and that Support should review?

If Support should be contacted anyway without preliminary forum discussion - then so be it.

Forwarded a link to this thread to Support.

yes Zheka is right.

I am going to help this time, but you folks have not the right attitude.

It takes a lot of time for me to collect stuff and write.

I have to say that Zorro's idea is nice, but a bug on bar aggregation and also the attitude of support shows that the product is not mature enough.

So,

If using the TimeFrame you got different. Try without Timeframe and change only BarPeriod. I spend more days with this problem. See, in my post (Indicator and TimeFrame).

For me is a Zorro best platform for developing AOS. I make more like 100 AOS under MT4 and 80% time I make workaround MT4.

I tried and with BarPeriod=5 it works as expected.

It seems TimeFrame formation simply begins from the first available bar in a data file. Timeframes have to be manually coded around anyway to work properly.

So what are the results? There is no bug? Only a question about the Timeframe?

Zorro is a great product! Many kudos for the team! There always be bugs. There is no bugs only if you don't develop anything. Saying that, it could be an area for improvement when it comes to quality (for example, Z3 system in the release version of Zorro, I've downloaded 3 release versions just to find out that Z3 isn't working again and again. Sometimes I think that QA doesn't check it at all. I hope I am wrong). As it was mentioned, better attitude will be beneficial for all and make Zorro even more powerful and of an excellent quality. I believe that it's a common goal for the Zorro's team and users.

JCL,

this is my communication with support that shows everything:


1. Why does a user need to get involved in the implementation of a software? What matters is if it works correctly.

2. To establish that, why can't Support do the quick QA check between MT4 and IB's data?

3. What do I do now after the last message?
- I indicated there is a problem
- I specified the process to replicate (compare to IB data)
- I went to MQL4 manual to confirm that MT4 in general works with bar open times AND that iTime works with bar open times...

IF this is not enough to get someone to simply download both IB's data and MT4's and do the checking firsthand - what else??
(who would blindly trust some files from a user anyway)

My understanding now is that there is a bar formation bug with MT4 plug-in.

I do not understand this problem. You claim to have data that proves a difference of IB and MT4. But you don't want to send it to us. We do not have such data and are not able to produce it, because all our MT4 and IB data contain totally different prices anyway. MT4 generates prices on its server and IB gets them from the market.

So I do not know why you cannot submit the details that we need for looking into the matter. Other users have no problem with that. You find a bug, send us the script or data, we reproduce it, we fix it, or find out that it's no bug. As simple as that. Happens all the time. What is your problem with this procedure? Lengthy discussions or manual interpretation speculations won't get any bug fixed, they will only get you to not being taken seriously anymore at some point.

The problem - again - is with the attitude.

I did not want to send the data this time as a matter of principle - because last time I did it (with IB plug-in), it was not taken seriously, and because this time it is really elementary.

IB's and MT4's data(from any STP/ECN broker) differ by only 0.1-0.3pip per side, but price direction is 100% the same.
So, comparing direction and range of bars with the same timestamp between sources will show if there is a problem...
Nothing mysterious. Just needs good, constructive attitude.

Secondly, Support believe(!) iTime function returns bar close time.
But MT4 works with bar open times. It is written in the MQL manual in numerous places, but also specifically for this function.

So, this looks like a possible reason for the error and should be checked and truth found, right?

Instead, I am asked to provide proof of communication with MQ support.
Attitude?


Messages from support like above, absence of QA, support refusing to read the forum - are not normal for people switching from other software.
In the same way, as Zorro's EndMarket=1900 excludes 19:00, while the whole world around (from exchanges to cafes) operates otherwise.

No speculation, only facts.

Broker: !Global Prime Pty Ltd

EURUSD M1 history from MT4:


Let's look at this tick:
2018.02.07,23:55,1.22700,1.22705,1.22683,1.22683,170

Now Zorro's history downloaded from this MT4 (time zone shift is 2h):


The same tick:
2018.02.07 21:55 1.227 1.22705 1.22683 1.22683 0 263.8

The timestamp is identical BUT Zorro uses bar close time while MT4 uses bar open time.
So, obviously it's a bug.

If there is a doubt that MT4 uses bar open time, well... here comes the proof:
Proof1:

Source: https://book.mql4.com/functions/datetime. Please read the full article if there are still doubts.

Proof2
Let's check iTime function definition but in Russian since MetaQuotes is a Russian company and their RnD is in Russia. I guess tutorial was translated from Russian to English. But in English version it's not directly said that it returns open bar time only indirectly. However, the Russian version explicitly says that it's an open bar time.
Russian version: https://docs.mql4.com/ru/series/itime
Translated with Google: link

Translated section with Google:


I forward the link to this post to Support.
Thank you!

Quick check of FXCM data shows the similar issue. FXCM has an open bar timestamp but Zorro treats it as a close bar timestamp. I assume you will be able to conduct a detailed examination if you need.

Maybe data from the other sources needs to be checked as well.

We received today several data files and logs from various users with bar issues. So, no need for Zheka's mystery data anymore.

No bar aggregation problem could be found with any of the received data files. So we must still conclude that Zorro bar aggregation is fine.

As to FXCM, it is clearly stated in the API manual that the time stamp is the bar start. However, we received today 2 different FXCM .t6 files, both also different to our own .t6 file, so it appears that the FXCM servers have slightly different history files. Maybe they are generating them local. We will look into this. Until then we will stick to treating FXCM time stamp as bar start.

As to MT4, when we developed the bridge we received a description by MetaQuotes how historical bars are generated. From this description the time stamp is the bar end. We'll contact them again for clarification. Inbetween, if you want to treat it as start time, just edit the Zorro.mq4 and add 60 to the iTime return value. Maybe we'll make this optional.

That's all info that I have about bar aggregation issues and worries.

I've continued my investigation of FXCM, MT4 and Zorro data.

FXCM provides a tool to download their data: FXCM Historical Data Downloder. It's possible to download both Bid and Ask M1 data (BTW maybe this explains different price history from FXCM: simply Bid and Ask data, more on this below).

As it's known Zorro uses ask prices and bar close time. FXCM uses bar open time.
Data downloaded from FXCM API with Zorro (Ask):
2018.02.07 21:57:00 1.22685 1.22686 1.22662 1.2267 119
2018.02.07 21:56:00 1.22704 1.22707 1.22685 1.22685 170
2018.02.07 21:55:00 1.22678 1.22705 1.22678 1.22704 151
2018.02.07 21:54:00 1.22666 1.22679 1.22663 1.22678 98
2018.02.07 21:53:00 1.22685 1.22696 1.22646 1.22666 301

Data downloaded from FXCM with their tool (Ask):
2/7/2018 21:57:00 1.2267 1.22673 1.22654 1.22655 119
2/7/2018 21:56:00 1.22685 1.22686 1.22662 1.2267 119
2/7/2018 21:55:00 1.22704 1.22707 1.22685 1.22685 170
2/7/2018 21:54:00 1.22678 1.22705 1.22678 1.22704 151
2/7/2018 21:53:00 1.22666 1.22679 1.22663 1.22678 98

Data looks good taking into account that FXCM uses open time and Zorro close time. Good!

Let's continue and compare FXCM bid price, FXCM MT4 price (bid as well) and Zorro's data downloaded from FXCM MT4.
Data downloaded from FXCM with their tool (Bid):
2/7/2018 21:53:00 1.22663 1.22676 1.2266 1.22675
2/7/2018 21:54:00 1.22675 1.22703 1.22675 1.22701
2/7/2018 21:55:00 1.22701 1.22705 1.22682 1.22682
2/7/2018 21:56:00 1.22682 1.22682 1.22662 1.2267
2/7/2018 21:57:00 1.2267 1.2267 1.2265 1.22652

Data exported from FXCM MT4 (bid as well cause MT4 uses bid):
2018.02.07 23:53 1.22662 1.22676 1.2266 1.22675
2018.02.07 23:54 1.22676 1.22702 1.22676 1.22701
2018.02.07 23:55 1.227 1.22705 1.22682 1.22682
2018.02.07 23:56 1.22682 1.22682 1.22662 1.22669
2018.02.07 23:57 1.2267 1.2267 1.22651 1.22652

Data almost identical (small differences less then PIP and also timezone shift: 2h). As we remember, FXCM uses bar open time. Data in MT4 is the same. So, it seems that MT4 uses bar open time as well.

Now data downloaded from FXCM MT4 with Zorro:
2018.02.07 21:53:00 1.22662 1.22676 1.2266 1.22675
2018.02.07 21:54:00 1.22676 1.22702 1.22676 1.22701
2018.02.07 21:55:00 1.227 1.22705 1.22682 1.22682
2018.02.07 21:56:00 1.22682 1.22682 1.22662 1.22669
2018.02.07 21:57:00 1.2267 1.2267 1.22651 1.22652

Data identical to FXCM's and MT4 data. But it should be identical because both FXCM and MT4 uses bar open time and Zorro bar close time.

I hope these results explain the issue.
To sum up: MT4 uses bar open time (as well as FXCM). Zorro downloads data from FXCM API correctly. Zorro's data downloaded from MT4 is shifted because of bar time confusion.

If you have any questions please feel free to ask!

gents, wouldn't it be simpler to print a M5 chart from Zorro and compare it visually to any other reliable source (like FXCM)?

If it does not look the same...

anyway, this is my last shot at it. Very last! attached the final evidence.

Luisd, I wonder how you downloaded data? Mine is different. I've just downloaded it with Zorro via FXCM plugin and it's inline with data downloaded directly from FXCM with their tool

And about the bar aggregation...
Let's check the source, the data. Charts are build based on data.

5m data downloaded directly from FXCM (timestamp is a bar open):
02-08-2018,12:50:00,1.2224,1.22269,1.22183,1.22223,2566
02-08-2018,12:55:00,1.22223,1.22371,1.22206,1.22363,2913
02-08-2018,13:00:00,1.22363,1.22398,1.22342,1.22381,2626
02-08-2018,13:05:00,1.22381,1.22535,1.22368,1.22509,3210
02-08-2018,13:10:00,1.22509,1.22548,1.22458,1.22517,2382

5m data aggregated by Zorro (timestamp is a bar close):
[1030: Thu 18-02-08 12:50] 1.2217/1.22251.2215/1.2224
[1031: Thu 18-02-08 12:55] 1.2224/1.22271.2218/1.2222
[1032: Thu 18-02-08 13:00] 1.2222/1.22371.2221/1.2236
[1033: Thu 18-02-08 13:05] 1.2236/1.22401.2234/1.2238
[1034: Thu 18-02-08 13:10] 1.2238/1.22541.2237/1.2251

I don't see any problems. You can check it by yourself but use the latest version of Zorro

The only thing that should be fixed is Zorro's shifted timestamps with MT4 data. I believe there is enough evidence for the Zorro's team. I wasn't able to find any other bugs...

kujo your data is indeed fine - can you post how you aggregated it?

I only aggregated it with BarPeriod = 5. No TimeFrame, no BarOffset and my M5 candles are clearly wrong

Hell... tell me how you did it... looking forward to your reply

I downloaded it via the Zorro Download.c script from FXCM too

No problem, pal.
It's just:

function run()
{
BarPeriod=5;
Verbose=3;
StartDate=20180208;
set(LOGFILE);
}

That's it.

The support got the FXCM data by luisd and it seemed indeed partially wrong. For some reason, the FXCM data seems to be sometimes wrong, sometimes correct. We're currently looking into the problem and will keep you informed.

downloaded from FXCM server GBDEMO with Zorro's Download.c
as you can see the M5 charts have the same overall shape but none of the candles are the same.

Can anyone please publish a 5M chart obtained with Zorro (FXCM feed) with BarPeriod = 5 -- thanks

Could you please check the date (modified date) of the FXCM plugin? Fxcm.dll in Plugins folder

AUG 2017 -- I am going to re-install everything...

Yeap, that's the problem I wrote about to the support. Wrong FXCM plugin was included in the release version. Please download the release version again.

done now it works!