I created T1 and T6 files with Tick Data from Dukascopy as described here: http://statsmage.com/zorro-on-steroids-ii-data-conversion/ http://statsmage.com/zorro-on-steroids-ii-data-conversion/

The whole process took two weeks on my system and I had to upgrade my hard drive to store these CSV files. So you will basically save two weeks, maybe a hard drive and if you help to keep this torrent alive for everyone -> win-win situation...!


If you want it, fire up your BitTorrent client (qTorrent is a decent one without adware) and get it. You should be selective about what you download. Not many will want to backtest with a variable spread on T1 data and hack Zorro to do this, so maybe deselect this file: T1_Forex_History_2006-2017_Bid_hack.7z

All data adds up to around 50 GB (compressed by 70%!)


Grab the torrents here: https://www.dropbox.com/sh/rbgocbu2cwx2ur3/AADGQm38GHjckW_y5CRsfYOfa?dl=0

If you find any mistakes in the dataset, please post your findings here!

Readme:

All credits to statsmage.com!


Do yourself a favor and enable NTFS compression for your Zorro History folder! It will save you 40% disk space!

This data comes in different packages:

Dukascopy started recording Tick Data between January 2003 and the December 2005, depending on the currency pair. The datatypes are M1, Tick Data Ask (Zorro default) and Tick Data Bid. The later can empower you to backtest on Tick Data with variable Spread. Details on statsmage.com or in the attached PDF "Zorro on Steroids III - Backtesting with Variable Spread.pdf"

The different torrent packages are:

-Tick Data Ask and Bid from January 2003 to December 2005 (some currencies records start in 2003 and some in 2005
-Tick Data Ask from January 2006 to December 2017
-Tick Data Bid from January 2006 to December 2017
-Tick Data Ask and Bid from January 2018 to June 2018 (will be replaced in January 2019 by the dataset of the complete year)
-M1 Data from January 2003 to December 2017
-M1 Data from January 2018 to June 2018 (will be replaced in January 2019 by the dataset of the complete year)

Included are all Major Forex Pairs and all Crosses (all possible combinations of AUD, CAD, CHF, EUR, GBP, JPY, NZD, USD)


Read the attached PDF file "Zorro on Steroids II - Data Conversion"
This thread might be interesting too: http://www.opserver.de/ubb7/ubbthreads.php?ubb=showflat&Number=463658

If you can afford the disk space and bandwidth, please don't delete the torrent and help to upload. I'm not willing to lever a potential several hundred gigabytes of upload from my private internet connection. -> Don't be a Leecher - become a Seeder and be awesome! ;-)

If you want to create these files, install python 2 from within the Cygwin setup. It didn't work for me when I installed it directly!

The original script published on http://statsmage.com/ works only with Zorro S. I edited the original script a little so you can use Zorro free in a two - steps process (or rather 3 steps if you count Tickdownloader in)

Set the time zone to (UTC) Coordinated Universal Time. Tickdownloader will export two identical CSV files (for example EURUSD_tick.csv and EURUSD_tick_UTC+0_00.csv). If you forgot this - the first file will contain the UTC time without daylight savings. You can easily delete the second file to save space during the process. Check "Automatic export to CSV" and only export tick data. The conversion to M1 will be done by the Zorro script.


Call the python script by opening the Cygwin Terminal. Enter "cd /cygdrive/d/zorro/strategy/dataconversion" if the python script is in D:ZorroStrategyDataconversion (without the quotes!) and press Enter.

Enter "python convert_all_assets.py /cygdrive/e/tickdata/" if your CSV files are in E:tickdata

Don't press any keys while the Cygwin terminal is active and a script is running. It might break the whole process!

If you want to optimally use your computer's resources, distribute the CSV files on different folders (as many as you have physical CPU Cores) and if possible on different physical hard drives as well, as the reversion step is limited by your hard drive's speed. An SSD really rocks here.

Edit the the ConvertData_all_assets.c script according to your needs (first couple of lines)

I do not use tick data, but great job and great sharing!

You can select only T6 Data (M1) as well! It's an extra archive within the torrents.

Thank you !!!!!!!

I am gonna seed it al least till the end of year 2018. If you create new torrent with 2018 year I am gonna seed it as well.

Thank youu!!!!

2018 complete!

https://www.dropbox.com/sh/rbgocbu2cwx2ur3/AADGQm38GHjckW_y5CRsfYOfa?dl=0

Awesome!

Thank you so much.. sdh309795gaas

I can't load the Torrent with UTorrent. It gives an error message about an invalid "Bencoding"

Try qBittorrent. The torrent seems okay. I have already uploaded the package 1,76 times.

Hi,

first of all - great effort!

I am interesed in the M1 data for forex crosses. Maybe I have missed something obvious (sorry if I have) but how to you update the assets.fix file with correct spread, price, etc?

Thanks!

Thanks!

My broker is FXChoice (MT4).
I just created the attached AssetsFix.csv with this script:

Many thanks!

I just found out that the dataset has some gaps. AUD/CAD is really bad in 2008. Use the script below to detect gaps. I commented it out to check for other gaps:

StartDate = 20060101;
EndDate = 20181231;

->

Asset Gaps
AUDJPY 4
EURGBP 5
GBPCHF 5
USDCHF 5
CHFJPY 6
EURUSD 6
GBPUSD 6
USDCAD 6
EURCHF 7
GBPJPY 7
NZDUSD 14
AUDNZD 19
EURJPY 23
USDJPY 24
AUDUSD 58
EURCAD 81
EURAUD 94
GBPAUD 119
AUDCHF 142
EURNZD 167
CADJPY 168
NZDCAD 181
GBPCAD 183
NZDCHF 187
CADCHF 193
NZDJPY 197
GBPNZD 221
AUDCAD 6263

Optimized asset loop:

while(asset(loop(
"AUD/JPY","EUR/GBP","GBP/CHF","USD/CHF","CHF/JPY","EUR/USD","GBP/USD",
"USD/CAD","EUR/CHF","GBP/JPY","NZD/USD","AUD/NZD","EUR/JPY","USD/JPY",
"AUD/USD","EUR/CAD","EUR/AUD","GBP/AUD","AUD/CHF","EUR/NZD","CAD/JPY",
"NZD/CAD","GBP/CAD","NZD/CHF","CAD/CHF","NZD/JPY","GBP/NZD","AUD/CAD")))

Hi sdh309795gaas,


It seems tickdownloader has been upgraded to latest Quant data manager.

The format of the data has been changed.

Would you have plan to update the script in http://statsmage.com/zorro-on-steroids-ii-data-conversion/ to cater the changes accordingly ?

As Zorro has a python bridge now, I will try to create a hassle free solution to update / create t1 and t6 files with just a few clicks.

I plan to share the complete 2019 package in January.

By the way, I'm not the guy behind statsmage!

Great to hear that

Okay, as 2019 is completed, it's time for the next torrent. However, something must have changed with Python and Cygwin.

I'm busy with other problems at the moment. If somebody creates a zorro script that does the job, I will release the new package immediately.

Anyone has 2019 data without gaps?
I have 11 months nothing

News on 2019 data?

My 2019 Data from the broker is really bad...

So here is the update:

I will share the data with Resilio Sync from now on, because it's much more convenient for me to release new data (I plan to do that every 3 months). When you installed the client, just
klick here or directly give your client this link: https://link.resilio.com/#f=ZorroT6...L4H6NL5LGBCWQIZLZPO3JJ&v=2.6&a=2

I will start with the T6 data since the last release and then add all previous T6 data later. I will stop seeding the old torrents in a couple of weeks.

At the moment, something isn't working with the conversion from tickdata to T1. I couldn't find the error at a quick glance and I don't plan to fix it any time soon, as I don't need tickdata. If somebody will fix it, I will release tickdata. The scripts for exporting are included in the shared folder.

Strategyquants QuantDataManager which I'm using to get the tick data from Dukascopy isn't able to export the (average?) spread anymore. This is why the dynamic live spread isn't available anymore for all T6 files after 2018.

You need 7-Zip to decompress the files.

Thank you so much, up and running

I just updated all T6 Data!

Updated!

Thank you Smon! I can't get Resilio to work though.. windows firewall permissions, etc. I can't find any good guide to resolve or fix the firewall issues.

I had issues with Resilio behind a proxy, but never behind a normal home router. If you give me a link to upload, maybe Nextcloud...

2020 Data added. Enjoy!

Hi Smon,
I read your post regarding Dukascopy data. Thanks for the effort you put into this issue. One question: do you still have t1 data from Dukascopy?

I would like to backtest Broker arbitrage, therefor I need a second brokers tick data (I already have FXCM t1 history from the major pairs). Would appreciate that.

Thanks in advance for your answer.

Simon

M1 Data until May 2021 added.

Thanks a lot for your effort!
Would be great if index/commodity cfds could be added to the archive.

M1 Data until Sep 2021 added.

Sorry, but I don't have plans to include index/commodity cfds.

Year 2021 complete. Enjoy!

Hello Smon,
you did a great job!!

Thank you.

You're welcome!

I just uploaded the new history until 9th April!

Sorry for the long time without an update - I uploaded all new data until 6th Feb 2023 yesterday!

Thank you, you are so generous to share your data with all of us, Smon - I also think similar to you:
When having done a long task, why not share it with others?
If more people would be like you, the world would be a better place !

The only problem now is that I did download the torrents and a client and that torrent seems to be dead now - 5years after it started.
So if anyone knows about other sources of such data, please revive this thread and share yours.
Thank you.

I just updated the files! For everybody, who is new to this:

@YesNotes, you are right! I stopped the torrent on my side and didn't provide the link to the ResilioSync folder. Sorry for that! I must have assumed, that nobody is really interested in the TickData. So here ist the ResilioSync link. Please note that I stopped releasing Tickdata after 2018, as I don't need it for myself.

https://link.resilio.com/#f=Zorro%2...6BNTQJYADRCX5QYRRQJ2VJ&v=2.7&a=2