Dukascopy T1&T6 files for all Forex Pairs and Crosses - get it!

Posted By: Smon

Dukascopy T1&T6 files for all Forex Pairs and Crosses - get it! - 06/27/18 19:39

I created T1 and T6 files with Tick Data from Dukascopy as described here: http://statsmage.com/zorro-on-steroids-ii-data-conversion/ http://statsmage.com/zorro-on-steroids-ii-data-conversion/

The whole process took two weeks on my system and I had to upgrade my hard drive to store these CSV files. So you will basically save two weeks, maybe a hard drive and if you help to keep this torrent alive for everyone -> win-win situation...!


If you want it, fire up your BitTorrent client (qTorrent is a decent one without adware) and get it. You should be selective about what you download. Not many will want to backtest with a variable spread on T1 data and hack Zorro to do this, so maybe deselect this file: T1_Forex_History_2006-2017_Bid_hack.7z

All data adds up to around 50 GB (compressed by 70%!)


Grab the torrents here: https://www.dropbox.com/sh/rbgocbu2cwx2ur3/AADGQm38GHjckW_y5CRsfYOfa?dl=0

If you find any mistakes in the dataset, please post your findings here!

Readme:

All credits to statsmage.com!


Do yourself a favor and enable NTFS compression for your Zorro History folder! It will save you 40% disk space!

This data comes in different packages:

Dukascopy started recording Tick Data between January 2003 and the December 2005, depending on the currency pair. The datatypes are M1, Tick Data Ask (Zorro default) and Tick Data Bid. The later can empower you to backtest on Tick Data with variable Spread. Details on statsmage.com or in the attached PDF "Zorro on Steroids III - Backtesting with Variable Spread.pdf"

The different torrent packages are:

-Tick Data Ask and Bid from January 2003 to December 2005 (some currencies records start in 2003 and some in 2005
-Tick Data Ask from January 2006 to December 2017
-Tick Data Bid from January 2006 to December 2017
-Tick Data Ask and Bid from January 2018 to June 2018 (will be replaced in January 2019 by the dataset of the complete year)
-M1 Data from January 2003 to December 2017
-M1 Data from January 2018 to June 2018 (will be replaced in January 2019 by the dataset of the complete year)

Included are all Major Forex Pairs and all Crosses (all possible combinations of AUD, CAD, CHF, EUR, GBP, JPY, NZD, USD)


Read the attached PDF file "Zorro on Steroids II - Data Conversion"
This thread might be interesting too: http://www.opserver.de/ubb7/ubbthreads.php?ubb=showflat&Number=463658

If you can afford the disk space and bandwidth, please don't delete the torrent and help to upload. I'm not willing to lever a potential several hundred gigabytes of upload from my private internet connection. -> Don't be a Leecher - become a Seeder and be awesome! ;-)

If you want to create these files, install python 2 from within the Cygwin setup. It didn't work for me when I installed it directly!

The original script published on http://statsmage.com/ works only with Zorro S. I edited the original script a little so you can use Zorro free in a two - steps process (or rather 3 steps if you count Tickdownloader in)

Set the time zone to (UTC) Coordinated Universal Time. Tickdownloader will export two identical CSV files (for example EURUSD_tick.csv and EURUSD_tick_UTC+0_00.csv). If you forgot this - the first file will contain the UTC time without daylight savings. You can easily delete the second file to save space during the process. Check "Automatic export to CSV" and only export tick data. The conversion to M1 will be done by the Zorro script.


Call the python script by opening the Cygwin Terminal. Enter "cd /cygdrive/d/zorro/strategy/dataconversion" if the python script is in D:ZorroStrategyDataconversion (without the quotes!) and press Enter.

Enter "python convert_all_assets.py /cygdrive/e/tickdata/" if your CSV files are in E:tickdata

Don't press any keys while the Cygwin terminal is active and a script is running. It might break the whole process!

If you want to optimally use your computer's resources, distribute the CSV files on different folders (as many as you have physical CPU Cores) and if possible on different physical hard drives as well, as the reversion step is limited by your hard drive's speed. An SSD really rocks here.

Edit the the ConvertData_all_assets.c script according to your needs (first couple of lines)
Posted By: MatPed

Re: Dukascopy T1&T6 files for all Forex Pairs and Crosses - get it! - 06/28/18 11:54

I do not use tick data, but great job and great sharing!
Posted By: Smon

Re: Dukascopy T1&T6 files for all Forex Pairs and Crosses - get it! - 06/28/18 17:51

You can select only T6 Data (M1) as well! It's an extra archive within the torrents.
Posted By: oligodar

Re: Dukascopy T1&T6 files for all Forex Pairs and Crosses - get it! - 07/04/18 14:33

Thank you !!!!!!!
Posted By: oligodar

Re: Dukascopy T1&T6 files for all Forex Pairs and Crosses - get it! - 07/04/18 14:50

I am gonna seed it al least till the end of year 2018. If you create new torrent with 2018 year I am gonna seed it as well.

Thank youu!!!!
Posted By: Smon

Re: Dukascopy T1&T6 files for all Forex Pairs and Crosses - get it! - 02/16/19 17:02

2018 complete!

https://www.dropbox.com/sh/rbgocbu2cwx2ur3/AADGQm38GHjckW_y5CRsfYOfa?dl=0
Posted By: Brax

Re: Dukascopy T1&T6 files for all Forex Pairs and Crosses - get it! - 02/16/19 17:13

Awesome!
Posted By: OptimusPrime

Re: Dukascopy T1&T6 files for all Forex Pairs and Crosses - get it! - 02/17/19 15:38

Thank you so much.. sdh309795gaas

I can't load the Torrent with UTorrent. It gives an error message about an invalid "Bencoding"
Posted By: Smon

Re: Dukascopy T1&T6 files for all Forex Pairs and Crosses - get it! - 02/17/19 19:18

Try qBittorrent. The torrent seems okay. I have already uploaded the package 1,76 times.
Posted By: j45p41

Re: Dukascopy T1&T6 files for all Forex Pairs and Crosses - get it! - 02/24/19 22:26

Hi,

first of all - great effort!

I am interesed in the M1 data for forex crosses. Maybe I have missed something obvious (sorry if I have) but how to you update the assets.fix file with correct spread, price, etc?

Thanks!
Posted By: Smon

Re: Dukascopy T1&T6 files for all Forex Pairs and Crosses - get it! - 02/26/19 18:57

Thanks!

My broker is FXChoice (MT4).
I just created the attached AssetsFix.csv with this script:

Code:
// connect to your broker and click the "Trade" button.
// this script creates this file: /zorro/log/assets.csv
// replace your assetsfix.csv in /zorro/history/ with this file!
// (better make a backup of the old one, jsut in case you need it again!)

function run()
{
	StartDate = 20180101;
	EndDate = 20181231;
	LookBack = 1;
	BarPeriod = 1440;


	while(asset(loop("AUD/CAD","AUD/CHF","AUD/JPY","AUD/NZD","AUD/USD","CAD/CHF","CAD/JPY","CHF/JPY","EUR/AUD","EUR/CAD","EUR/CHF","EUR/GBP","EUR/JPY","EUR/NZD","EUR/USD","GBP/AUD","GBP/CAD","GBP/CHF","GBP/JPY","GBP/NZD","GBP/USD","NZD/CAD","NZD/CHF","NZD/JPY","NZD/USD","USD/CAD","USD/CHF","USD/JPY"))) 
									
	{
		vars Price = series(price());
	}
}



Attached File
AssetsFix.csv  (8 downloads)
Posted By: OptimusPrime

Re: Dukascopy T1&T6 files for all Forex Pairs and Crosses - get it! - 02/27/19 13:50

Many thanks!
Posted By: Smon

Re: Dukascopy T1&T6 files for all Forex Pairs and Crosses - get it! - 04/22/19 12:14

I just found out that the dataset has some gaps. AUD/CAD is really bad in 2008. Use the script below to detect gaps. I commented it out to check for other gaps:

Code:
function run()
{
	StartDate = 20080101;
	EndDate = 20120101;
	
	BarPeriod = 15;
	
	//"AUD/CAD",
	while(asset(loop("AUD/CHF","AUD/JPY","AUD/NZD","AUD/USD","CAD/CHF","CAD/JPY",
									"CHF/JPY","EUR/AUD","EUR/CAD","EUR/CHF","EUR/GBP","EUR/JPY","EUR/NZD",
									"EUR/USD","GBP/AUD","GBP/CAD","GBP/CHF","GBP/JPY","GBP/NZD","GBP/USD",
									"NZD/CAD","NZD/CHF","NZD/JPY","NZD/USD","USD/CAD","USD/CHF","USD/JPY"))) 
									
	{
		
		var tdiff = (wdate() - wdate(1)); //time difference in days of the last consecutive bars

		//search for time differences at 1.5 times longer than BarPeriod:
		if(tdiff > (BarPeriod/1440)*1.5 && tdiff < 1) //1 day = 1440 minutes....
		{
			printf("n%d-%02d-%02d %02d:%02d        %s gap: %.0f minutes missing",
				year(), month(), day(), hour(), minute(), Asset, (wdate() - wdate(1))*1440);
		}
	}
}

Posted By: Smon

Re: Dukascopy T1&T6 files for all Forex Pairs and Crosses - get it! - 04/22/19 13:18

StartDate = 20060101;
EndDate = 20181231;

->

Asset Gaps
AUDJPY 4
EURGBP 5
GBPCHF 5
USDCHF 5
CHFJPY 6
EURUSD 6
GBPUSD 6
USDCAD 6
EURCHF 7
GBPJPY 7
NZDUSD 14
AUDNZD 19
EURJPY 23
USDJPY 24
AUDUSD 58
EURCAD 81
EURAUD 94
GBPAUD 119
AUDCHF 142
EURNZD 167
CADJPY 168
NZDCAD 181
GBPCAD 183
NZDCHF 187
CADCHF 193
NZDJPY 197
GBPNZD 221
AUDCAD 6263

Optimized asset loop:

while(asset(loop(
"AUD/JPY","EUR/GBP","GBP/CHF","USD/CHF","CHF/JPY","EUR/USD","GBP/USD",
"USD/CAD","EUR/CHF","GBP/JPY","NZD/USD","AUD/NZD","EUR/JPY","USD/JPY",
"AUD/USD","EUR/CAD","EUR/AUD","GBP/AUD","AUD/CHF","EUR/NZD","CAD/JPY",
"NZD/CAD","GBP/CAD","NZD/CHF","CAD/CHF","NZD/JPY","GBP/NZD","AUD/CAD")))
Posted By: aventador

Re: Dukascopy T1&T6 files for all Forex Pairs and Crosses - get it! - 11/17/19 13:45

Originally Posted by sdh309795gaas
Try qBittorrent. The torrent seems okay. I have already uploaded the package 1,76 times.



Hi sdh309795gaas,


It seems tickdownloader has been upgraded to latest Quant data manager.

The format of the data has been changed.

Would you have plan to update the script in http://statsmage.com/zorro-on-steroids-ii-data-conversion/ to cater the changes accordingly ?
Posted By: Smon

Re: Dukascopy T1&T6 files for all Forex Pairs and Crosses - get it! - 11/30/19 14:32

As Zorro has a python bridge now, I will try to create a hassle free solution to update / create t1 and t6 files with just a few clicks.

I plan to share the complete 2019 package in January.

By the way, I'm not the guy behind statsmage!
Posted By: aventador

Re: Dukascopy T1&T6 files for all Forex Pairs and Crosses - get it! - 12/13/19 20:51

Originally Posted by sdh309795gaas
As Zorro has a python bridge now, I will try to create a hassle free solution to update / create t1 and t6 files with just a few clicks.

I plan to share the complete 2019 package in January.

By the way, I'm not the guy behind statsmage!


Great to hear that
Posted By: Smon

Re: Dukascopy T1&T6 files for all Forex Pairs and Crosses - get it! - 01/02/20 19:18

Okay, as 2019 is completed, it's time for the next torrent. However, something must have changed with Python and Cygwin.

I'm busy with other problems at the moment. If somebody creates a zorro script that does the job, I will release the new package immediately.
Posted By: danatrader

Re: Dukascopy T1&T6 files for all Forex Pairs and Crosses - get it! - 02/22/20 09:41

Anyone has 2019 data without gaps?
I have 11 months nothing frown
Posted By: danatrader

Re: Dukascopy T1&T6 files for all Forex Pairs and Crosses - get it! - 02/25/20 21:19

News on 2019 data?

My 2019 Data from the broker is really bad...
Posted By: Smon

Re: Dukascopy T1&T6 files for all Forex Pairs and Crosses - get it! - 04/14/20 06:28

So here is the update:

I will share the data with Resilio Sync from now on, because it's much more convenient for me to release new data (I plan to do that every 3 months). When you installed the client, just
klick here or directly give your client this link: https://link.resilio.com/#f=ZorroT6...L4H6NL5LGBCWQIZLZPO3JJ&v=2.6&a=2

I will start with the T6 data since the last release and then add all previous T6 data later. I will stop seeding the old torrents in a couple of weeks.

At the moment, something isn't working with the conversion from tickdata to T1. I couldn't find the error at a quick glance and I don't plan to fix it any time soon, as I don't need tickdata. If somebody will fix it, I will release tickdata. The scripts for exporting are included in the shared folder.

Strategyquants QuantDataManager which I'm using to get the tick data from Dukascopy isn't able to export the (average?) spread anymore. This is why the dynamic live spread isn't available anymore for all T6 files after 2018.

You need 7-Zip to decompress the files.
Posted By: danatrader

Re: Dukascopy T1&T6 files for all Forex Pairs and Crosses - get it! - 04/15/20 20:55

Thank you so much, up and running laugh
© 2020 lite-C Forums