Is there an "official" way we can get access to more history data? I don't mean more history of what we already have on your Download page, but specifically more pairs.

For example IBFX has at least 27 tradeable pairs and I noticed FXCM has even more. In my discretionary trading, I found that some exotics just don't seem to work well, but others work great. I'd love to throw Zorro at them and see what it can uncover...

I'm willing to do the heavy lifting and gather/process/sort this data, to make more available for other users to easily Download. In the past when I've manipulated tickdata I've taken the raw source (usually .csv) and imported it into a MySQL table for analysis/cleanup. In this form it can also be (relatively easily) shifted to Zorro's UTC upon export and non-essential info stripped.

My biggest question for you is regarding licensing / legality restrictions. If we assume that the data itself is generally clean/error-free... is there still any issue with providing the data back in a cleaner format ready for import to Zorro?

I've found several "free providers" of data. I'd greatly prefer to use FXCM's data (mostly since it's a known trustworthy source) but it seems you have to purchase their data downloader tool which I'm unable to do. Also, I'm not sure if they have restrictions on redistribution.

One free and comprehensive provider is http://www.histdata.com/ but my impression here is that this is not "legally redistributed" data.

There are many other providers as well.

Using Zorro for downloading data from FXCM is free and covered by the API license.

The loadHistory function only loads M1 data at the moment, but if this is desired, we can also implement a way to download tick data.

Problem is that other broker's data is not free. And you can most likely not download any data from MT4 brokers, due to the limited price history of MT4 servers.

Awesome, I did not know you were covered for all FXCM data! I'm not so concerned about tickdata right now, but I might be someday.

FXCM data is fine I think for "low frequency trading" as you say. This is great!!

dusktrader, if you get interested in tick data again, Dukascopy has a comprehensive tick archive back to 2007 for a lot of pairs. Enter `php dukascopy' in google, and click on the first link to obtain software needed to download the data. It is updated every hour.

I suppose I'm glad not to be the only one messing with data here and there.

Although, I've so far used tick data only to synthesize various interesting bar formations (constant range bars, renko, median renko). And only played with EUR/USD, which is currently 844MB (2007-2013) heavily compressed, just to give you an idea of how much space tick data requires.

Well technically the only reason I "mess with data" is because I know how lol (read: I would prefer not to, but having access to good data is a necessary evil to build a robot!)

Thanks for the dukascopy info, I will definitely keep that in mind. I would prefer to try the FXCM route first since that data is officially licensed (in other words, the effort we invest in formatting FXCM data could be repurposed for the Zorro community I think, whereas maybe not with dukascopy data due to licensing).

Now if someone wants to make an argument that XX-vendor's data is better/cleaner/superior to the FXCM data, then I'm interested to hear about it for sure. I know there are holes in all data (I've heard), so I think the best procedure might be to run analysis on each group of downloaded data and be aware of how significant those holes may be.

I've done some limited manipulation of tickdata in the past but not ready to go there yet (I have so much to play with already being the Zorro n00b that I am). Some interesting research I've done before with synthesizing tickdata included grabbing different segments of multiple price streams and sewing them together in a normalized fashion (ie, take a week from EURUSD and sew it to the next week of GBPUSD, normalized, for example). The intent with that research was to try to improve robustness of the strategy, but also because the testing platform did not (at the time) support multiple currencies at once I think.

Anyway, I'm sure there are a million directions we could go... I'd rather start with getting a full library of (in my case) the 27 pairs under my belt. That in and of itself will keep me busy for awhile with Zorro...

Yeah, that sound fun. I've even seen somewhere that people created some artifical forex index which is more tradable than it's components. By the moto, if you can't create profitable strategy for the market, create market that suits your strategy. I like that idea.

And I like that you're thinking out of the box, just like I'm trying to do occassionaly. Stick with Zorro, it's a great tool. And most importantly have fun!

Zorro, at least in concept from what I've read, is the most comprehensive and robust platform for building an automated trading strategy I've ever seen (that I can afford LOL). I can't wait to prove that as I invest more time in understanding how it works.

The way I see it, if you aren't a big institution with unlimited pockets for R&D, then you have to really REALLY optimize your thought process. Since time itself is a limited resource for everyone, it makes sense to align yourself with a platform that already has all the heavy-lifting figured out! By heavy-lifting, I am referring to the math, analysis, optimization theories, and of course the actual process of trading the account (buy/sell/stop/etc). These are stumbling blocks that essentially waste time (in my opinion) with other platforms like Metatrader. (Not to mention the limitation of Metatrader in the big picture-- for example, what if you do manage to find success with that but then want to take that success to the next level like Interactive Brokers platform... do you start all over from scratch then?)

The platform they have here called Zorro has all the heavy-lifting already figured out, so as a trader we can focus more time on the actual strategy part.

I agree with you on all accounts. :thumbsup:

I'm working on loadHistory to download/update additional pairs into Zorro and I have a couple questions please:

I'm using mostly the example script from the manual for loadHistory

It seems that you need actual FXCM credentials to access their API. Can you please confirm that using a "Real" account pulls from a different datafeed than when selecting "Demo" in Zorro? As I recall FXCM does publish separate datafeeds (or maybe that was for segregating Standard and Mini? I can't remember). Fortunately I do have a dormant FXCM real account and it allowed it to work.

At first it wasn't working, with an error message similar to the following:

I discovered that I need to delete the NZDUSD_2013.bar file first (presumably that contained remnants from IBFX when I setup the Assets.dta file?)

In any case, it seemed to start working after ensuring there was no bar file:

After some time, it reported that it was done, but it seems that it only created a NZDUSD_2013.bar file. Is there some limit on the amount of data that can be requested -- do you know? I am requesting M1 data

Aside from the report Zorro gave for missing data during this download (gap)... is there any other way I would know this? There should be a way to check data integrity before running a backtest. For example it would be nice if Zorro could run an integrity check on data upon request and/or have a way to deal with gaps when backtesting.

Additionally, I wanted to report that it seems to not be honoring the NumYears setting. In trying this again, I can see that it downloads the entire 2013 (backwards from present to Jan 1st). But then it stops and doesn't try the prior years.

Here is the script I'm using:


EDIT: if I force the range with
StartDate = 20120101;
EndDate = 20121231;
then it downloads "something" but only a very small amount.

Could this issue be that FXCM doesn't provide comprehensive data on NZD/USD? or do you think this is some other glitch? Is there a way to turn on a debug to see what data it is actually getting and/or any messages from FXCM?

Thanks