Gamestudio Links
Zorro Links
Newest Posts
Mixed frequency history (M1+D)
by jcl
Today at 12:21
New Zorro version 2.12
by jcl
Today at 11:48
This is my fault-tolerant ATR indicator for you
by OptimusPrime
05/18/19 16:35
missing trade variables
by AndrewAMD
05/18/19 12:37
export asset to .txt file
by sdh309795gaas
05/17/19 16:21
LookBack mysteries
by sdh309795gaas
05/17/19 06:28
It is easy for you.
by AndrewAMD
05/16/19 16:39
AUM Magazine
Latest Screens
The Space Between
Pogostuck: Rage With Your Friends
Worst Case Z
AckCon'18 - Lotter vs the World 2 - Preview Release
Who's Online
11 registered (AndrewAMD, kvm, jumpman, mk_1, Dalla, MatPed, FLD, Realspawn, JeyKey II, 1 invisible), 460 Guests and 6 Spiders online.
Key: Admin, Global Mod, Mod
Newest Members
h4h4rd, ccm, Pippo, hurleydood, JohnLouderback
18222 Registered Users
Page 1 of 3 1 2 3 >
Topic Options
Rate This Topic
#459580 - 05/26/16 16:00 Backtesting with T1 data and variable spread *****
pcz Offline
Member

Registered: 05/17/16
Posts: 180
Loc: Prague
Hello everyone!

From the beginning I needed to backtest with tick data collected from my own broker that has variable spread. However I wasn't able to find how to do that. It took me several days but I think that I got it right finally. I will write the steps I took so it can serve as a tutorial for other people attempting to do something similar. I also hope that this will eventually help to discover bugs as I don't have much experience in C programming.

1. Obtaining data
For this demonstration I will use Dukascopy tick data downloaded using StrategyQuant Tick Downloader. First I downloaded the data for each currency pair and then I exported them to .csv files, each line in this format: YYYY-MM-DD hh:mm:ss.fff,bid,ask,bidVol,askVol

2. Splitting data
T1 conversion is more complicated than M1 conversion because there is no maximum size of the data set and the physical memory may not be sufficient to read the input file at once. Because I didn't want to end up wit too complicated Zorro conversion script I decided to split the data and reverse the line order using Python script and Linux shell command 'tac' (reversed 'cat' laugh ). The script can be found here: split.py. To use it in Windows you will need something like Cygwin.

3. Conversion
I had to modify the Zorro/Strategy/Convert.c script to accomodate for T1 structs ( ConvertT1.c ). You need to set 'base_path', 'smbl', 'start' and 'end' variables to use it. The input files should be in aforementioned CSV format (produced by split.py) and stored in base_path folder. Their names should be in this form: [SYMBOL]_[YEAR].csv. The important part is that if the SPREAD is defined using #define directive in the ConvertT1.c script, another .t1 file (besides the one with ask prices) is exported. Its name contains 's' at the end of symbol's name and it contains spread values for each tick.

4. Checking the converted data
To make sure that the conversion has been done properly you can use this modified export script: ExportT1.c

5. Simulating variable spread
The final step is using both .t1 files during backtest to simulate variable spread. One of the ways to do so is to include both SYMBOL and SYMBOLs in the backtest. SYMBOLs is an artificial asset used only to get the spread value for the actual asset. When the program loops through SYMBOLs, save the spread value from T1 struct to a global variable. When the program loops through the actual SYMBOL, assign the spread from the global variable to Zorro's Spread variable. And that's it laugh

Top
#459625 - 05/29/16 10:17 Re: Backtesting with T1 data and variable spread [Re: pcz]
pcz Offline
Member

Registered: 05/17/16
Posts: 180
Loc: Prague
This is probably question for Zorro developers...

So...I am setting the spread as described above. However I'm using the tick() and priceClose() to assign it to a global variable first. I have TICKS flag set.

Code:
float spread;
function tick()
{
   // if current asset == artificial spread asset
   if(!strcmp(Asset, ast1s))
      spread = priceClose();
}



From Zorro documentation: In a TMF or tick function, priceClose() returns the last price quote, updated every tick when new price data becomes available.

The prices seem to be OK in general but the spread values are sometimes weird. I double-checked that the values I'm storing in my T1 conversion script are fine. Is there any kind of smoothing or something that could cause this? Here's what I get when exporting the T1 values. I aligned the corresponding times in original and converted series and highlited the discrepancies.





A link to download the spreadsheet: https://we.tl/0ikhO2QXAZ

Top
#459696 - 05/31/16 18:50 Re: Backtesting with T1 data and variable spread [Re: pcz]
jcl Online

Chief Engineer

Registered: 07/22/00
Posts: 26866
Loc: Frankfurt
Zorro automatically fixes outliers. Probably that correction acts here since spread data always has lots of outliers. Set Detrend to NOPRICES for telling Zorro that this asset must not be fixed.

Top
#459809 - 06/07/16 14:33 Re: Backtesting with T1 data and variable spread [Re: jcl]
pcz Offline
Member

Registered: 05/17/16
Posts: 180
Loc: Prague
Thank you for the reply. One last question - how does the selection of the next T1 price work? I mean - obviously you can save more than 10 prices per second to the .t1 file. But only one price per every 100 ms is considered during the backtest. If I export the .t1 data to .csv file I see that the millisecond portion of selected price slowly changes, for example:

hh:mm:ss.x40
hh:mm:ss.x40
hh:mm:ss.x40
hh:mm:ss.x40
hh:mm:ss.x41
hh:mm:ss.x41
hh:mm:ss.x41
hh:mm:ss.x42
hh:mm:ss.x42
hh:mm:ss.x42
etc... (but much slower)

The reason I'm asking this is that I want to trim the data before I convert them to .t1 files so the resulting file is as small as possible. It's especially important with some brokers that have hundreds of prices per second. But there seems to be no clear split (like lets say Zorro would take the last price from each whole hundred ms: .100, .200, ...) - rather it seems that the split shifts slowly, one millisecond at a time. Is that correct?

Top
#459813 - 06/07/16 17:06 Re: Backtesting with T1 data and variable spread [Re: pcz]
jcl Online

Chief Engineer

Registered: 07/22/00
Posts: 26866
Loc: Frankfurt
As to my knowledge, there is no limit to the number of ticks, only to the number of bars. You can have only 10 bars per second, but any number of ticks.

Top
#459819 - 06/07/16 19:27 Re: Backtesting with T1 data and variable spread [Re: jcl]
pcz Offline
Member

Registered: 05/17/16
Posts: 180
Loc: Prague
Originally Posted By: jcl
As to my knowledge, there is no limit to the number of ticks, only to the number of bars. You can have only 10 bars per second, but any number of ticks.


You are absolutely right. I didn't realize that even with TICKS flag set run() is called on every bar and not every tick. Now, when outputting the prices in tick() function, I see them all with the exact timestamps +/- 1 ms. Thank you! laugh

Top
#459859 - 06/09/16 02:06 Re: Backtesting with T1 data and variable spread [Re: pcz]
boatman Offline
Senior Member

Registered: 04/23/14
Posts: 481
Loc: Sydney, Australia
pcz, thank you for sharing this. Extremely useful!
_________________________
www.robotwealth.com

Top
#463654 - 12/20/16 00:14 Re: Backtesting with T1 data and variable spread [Re: pcz]
CaptainChezza Offline
Newbie

Registered: 01/21/16
Posts: 12
Loc: New South Wales
Hey guys,
is this currently the best way to import tick data for zorro, or have things changed?
If I test with FXCM on zorro it just uses 1m OHLC; if I subscribe and get Zorro S, do I have access to reliable tick data on FXCM, or is it a bit hit and miss?

Thanks

Top
#463658 - 12/20/16 15:40 Re: Backtesting with T1 data and variable spread [Re: CaptainChezza]
pcz Offline
Member

Registered: 05/17/16
Posts: 180
Loc: Prague
You can download FXCM T1 data on Zorro Download page. The way I described is meant only for sources for which there is no possibility to download the data using Zorro or for situations in which it is necessary to backtest with variable spread. The variable spread backtesting can be achieved also with the new T6 struct as it allows for storing additional data besides OHLC. Related links:

http://zorro-trader.com/download.php

http://zorro-project.com/manual/en/export.htm


Edited by pcz (12/20/16 15:41)

Top
#463666 - 12/21/16 01:36 Re: Backtesting with T1 data and variable spread [Re: pcz]
CaptainChezza Offline
Newbie

Registered: 01/21/16
Posts: 12
Loc: New South Wales
Is FXCM T1 data that reliable, or is it worth using duka.
Variable spread is handy though.



Futher, I'm definitely a noob with this, but where do you include in the your py script the csv file name?

Also, do you then just drag and drop the py script into the cygwin terminal?

Code:
$ /cygdrive/d/Trading/DukaData/QmYybyXg.py
/cygdrive/d/Trading/DukaData/QmYybyXg.py: line 1: from: command not found
/cygdrive/d/Trading/DukaData/QmYybyXg.py: line 2: import: command not found
/cygdrive/d/Trading/DukaData/QmYybyXg.py: line 3: $'r': command not found
/cygdrive/d/Trading/DukaData/QmYybyXg.py: line 4: syntax error near unexpected t                  oken `sys.argv'
'cygdrive/d/Trading/DukaData/QmYybyXg.py: line 4: `if len(sys.argv) != 2:



I also tried using the CSVtoT6 script as I noticed in that script there has information about converting trade station csv into t1.
So I put my csv file name into that line:
Code:
#ifdef TRADESTATION
#define TCK T1
string InName = "HistoryEURCHF_UTC_Ticks_Bid_2015.02.01_2016.12.19.csv";  // name of the CSV file
string OutName = "Historyticks.t1";



But that didn't work. Is it meant to?
I am using the normal zorro, not zorro s. Do I need to have zorro s for this to function properly?

If I wanted to test T1 data on EURCHF for example can I create a fxcm account, and download it from there?
Or what is a suitable way to pull it from a CSV file from dukascopy? (without worrying about variable spread at this point in time)

Do I simply create a back the front CSV file divided into years, into the format of:
Date, value

Or does your python script do that?

I do apologise for jumping off topic a little (since i'm now just talking about tick imports)
Thanks


Edited by CaptainChezza (12/21/16 09:51)

Top
Page 1 of 3 1 2 3 >



Gamestudio download | chip programmers | Zorro platform | shop | Data Protection Policy

oP group Germany GmbH | Birkenstr. 25-27 | 63549 Ronneburg / Germany | info (at) opgroup.de