Gamestudio Links
Zorro Links
Newest Posts
zorro 64bit command line support
by 7th_zorro. 04/20/24 10:06
StartWeek not working as it should
by jcl. 04/20/24 08:38
Data from CSV not parsed correctly
by jcl. 04/20/24 08:32
Zorro FIX plugin - Experimental
by jcl. 04/20/24 08:30
folder management functions
by VoroneTZ. 04/17/24 06:52
lookback setting performance issue
by 7th_zorro. 04/16/24 03:08
Zorro FIX plugin - Experimental
by flink. 04/14/24 07:46
AUM Magazine
Latest Screens
The Bible Game
A psychological thriller game
SHADOW (2014)
DEAD TASTE
Who's Online Now
3 registered members (7th_zorro, Aku_Aku, 1 invisible), 579 guests, and 1 spider.
Key: Admin, Global Mod, Mod
Newest Members
EternallyCurious, howardR, 11honza11, ccorrea, sakolin
19047 Registered Users
Previous Thread
Next Thread
Print Thread
Rate Thread
Page 1 of 2 1 2
iVolatility options csv file to .t8 Zorro file #473093
06/12/18 22:24
06/12/18 22:24
Joined: May 2018
Posts: 134
S
SBGuy Offline OP
Member
SBGuy  Offline OP
Member
S

Joined: May 2018
Posts: 134
So, I decided to buy some real options data from iVolatility for my options backtesting.

However, there's no Zorro script for converting CSV to .t8 file. Anyone care to share a few lines of code on how I might do this?

CSVtoHistory only produces .t6 files.

Thanks.

Re: iVolatility options csv file to .t8 Zorro file [Re: SBGuy] #473094
06/13/18 00:32
06/13/18 00:32
Joined: Feb 2017
Posts: 1,725
Chicago
AndrewAMD Offline
Serious User
AndrewAMD  Offline
Serious User

Joined: Feb 2017
Posts: 1,725
Chicago
The manual is your friend! laugh

You can parse CSV to t8:
http://zorro-project.com/manual/en/data.htm

CSVtoHistory is a great template. So modify it to suit your needs.

Other resources:
http://zorro-project.com/manual/en/file_.htm
http://zorro-project.com/manual/en/str_.htm

Re: iVolatility options csv file to .t8 Zorro file [Re: AndrewAMD] #473106
06/13/18 23:05
06/13/18 23:05
Joined: May 2018
Posts: 134
S
SBGuy Offline OP
Member
SBGuy  Offline OP
Member
S

Joined: May 2018
Posts: 134
Well, I have been going to my friendly manual before posting for help. However, it is not being cooperative, or I'm still very rusty with coding.

Anyhow, I successfully made a T8 file, verified with History.c, but only after manually changing the format of the Expiration Date field in the CSV file using Excel. Trying to be less of a hacker, I would like to massage that field in my Zorro script.

The problem is that the Expiration Date string appears as 3/24/2017, where the month and day is sometimes 1 or 2 chars. I need to transpose this into an 8 digit long - YYYYMMDD, which is what the t8 file format requires.

I'm been hacking with strdate, wdatef, ymd, strf, etc. all day and can't seem to get to YYYYMMDD.

How would you do this Master Andrew? I'm sure you probably have a 1 liner for this problem :-)

Many thanks!


Last edited by SBGuy; 06/13/18 23:06.
Re: iVolatility options csv file to .t8 Zorro file [Re: SBGuy] #473108
06/13/18 23:27
06/13/18 23:27
Joined: May 2018
Posts: 134
S
SBGuy Offline OP
Member
SBGuy  Offline OP
Member
S

Joined: May 2018
Posts: 134
Wait.... Here is the solution!

ymd(wdatef("%m/%d/%Y",MyDate)

I'm going to post the full script when I'm done. No one should have to spend this much time writing a damn CSV import script, when whe should be spending time making money! :-)

Re: iVolatility options csv file to .t8 Zorro file [Re: SBGuy] #473109
06/13/18 23:31
06/13/18 23:31
Joined: Feb 2017
Posts: 1,725
Chicago
AndrewAMD Offline
Serious User
AndrewAMD  Offline
Serious User

Joined: Feb 2017
Posts: 1,725
Chicago
Yay! laugh

Re: iVolatility options csv file to .t8 Zorro file [Re: AndrewAMD] #473131
06/15/18 16:07
06/15/18 16:07
Joined: May 2018
Posts: 134
S
SBGuy Offline OP
Member
SBGuy  Offline OP
Member
S

Joined: May 2018
Posts: 134
I had some help from the guys at oP and they will post the final script on the manual. There were a lot of nuances that were not well documented when it comes to .t8 files.

Re: iVolatility options csv file to .t8 Zorro file [Re: SBGuy] #478412
10/16/19 20:59
10/16/19 20:59
Joined: Sep 2019
Posts: 3
P
Phrendo Offline
Guest
Phrendo  Offline
Guest
P

Joined: Sep 2019
Posts: 3
SBguy, I've looked all over for this script you worked on. I have the same conversion from IVol to tackle and can't seem to get it right. Did you end up posting this script anywhere?

Re: iVolatility options csv file to .t8 Zorro file [Re: SBGuy] #478502
10/30/19 22:06
10/30/19 22:06
Joined: May 2018
Posts: 134
S
SBGuy Offline OP
Member
SBGuy  Offline OP
Member
S

Joined: May 2018
Posts: 134
Sorry about the delay in response. Zorro posted it to their manual a while ago. It's at the bottom of the contract manual page.

https://zorro-project.com/manual/en/contract.htm

Code
// Example script for converting EOD options data to .t8:
// Format: underlying symbol, exchange, date MMDDYYYY, adj close, option symbol, expiry MMDDYYYY, strike, Call/Put, American/European, ask, bid, volume, open interest, close
// Sample: "TLT,NYSEArca,04/10/2015,129.62,TLT   150410C00112500,04/10/2015,112.5,C,A,17.3,16.2,0,0,129.62"
string Format = ",,%m/%d/%Y,,,i,f,s,s,f,f,f,f,f";

void main() 
{
// first step: parse the CSV file into a dataset
  int Records = dataParse(1,Format,FILENAME);
  printf("\n%d Records parsed",Records);
// second step: convert the raw data to the final CONTRACT format
  for(i=0; i<Records; i++,c++) 
  {
    CONTRACT* C = dataAppendRow(2,9);
    C->time = dataVar(1,i,0);
    string PC = dataStr(1,i,3);
    string EA = dataStr(1,i,4);
    C->Type = ifelse(*PC == 'P',PUT,CALL) + ifelse(*EA == 'E',EUROPEAN,0);
    int Expiry = dataInt(1,i,1); 
    C->Expiry = 10000*(Expiry%10000) + Expiry/10000; // MMDDYYYY -> YYYYMMDD
    C->fStrike = dataVar(1,i,2);
    C->fAsk = dataVar(1,i,5);
    C->fBid = dataVar(1,i,6);
    C->fVol = dataVar(1,i,7);
    C->fVal = dataVar(1,i,8); // open interest
    C->fUnl = dataVar(1,i,9);
    if(!progress(100*i/Records,0)) break; // show a progress bar
  }
  dataSort(2);
  dataSave(2,"History\\MyOptions.t8");
}



Re: iVolatility options csv file to .t8 Zorro file [Re: SBGuy] #478787
12/18/19 09:24
12/18/19 09:24
Joined: Dec 2019
Posts: 6
I
interzonez Offline
Newbie
interzonez  Offline
Newbie
I

Joined: Dec 2019
Posts: 6
How to run this script on my CSV file of historical SPX options data?

Where to point to my CSV file or does it run in the same directory somehow?

Thank you.

laugh I see FILENAME - assuming I can swap out here for the CSV file?

Last edited by interzonez; 12/18/19 09:28.
Re: iVolatility options csv file to .t8 Zorro file [Re: SBGuy] #478793
12/20/19 14:02
12/20/19 14:02
Joined: May 2018
Posts: 134
S
SBGuy Offline OP
Member
SBGuy  Offline OP
Member
S

Joined: May 2018
Posts: 134
Yes. And output t8 is the last line in the script.

You might also need to play with Format string depending on the specific CSV format/order of your incoming data. See the manual on Format. Once you master that, you can import just about any CSV data into Zorro t1, t6, or t8 format.

I believe the latest distro of Zorro includes a variation of this script as an included script now

Re: iVolatility options csv file to .t8 Zorro file [Re: SBGuy] #478797
12/21/19 03:11
12/21/19 03:11
Joined: Dec 2019
Posts: 6
I
interzonez Offline
Newbie
interzonez  Offline
Newbie
I

Joined: Dec 2019
Posts: 6
Thanks, yep using the script from the latest distro. Going around and around trying to match the format string to the CSV.

script and one line of my csv. probably I am missing something very simple and fundamental!

It parses 340648 SPX records. Then the error is: Error 013 invalid parameter 0

Date Expiry Under Strike Bid Ask OI Imp. vol fwd price Type
05/28/2019 12/20/2019 SPX C 100 2672.20 2675.80 533 1.696005 2810.23 E

Code

int Year = 2019;
string Ticker = "SPX"; // File name: SPX_2019.csv
string Format = "%m/%d/%Y,i,s,s,f,f,f,f,f,f,s"; // from the sample above

void main() 
{
// first step: parse the CSV file into a dataset
	int Records = dataParse(1,Format,strf("History\\%s_%i.csv",Ticker,Year));
	printf("\n%d %s Records parsed",Records,Ticker);
	if(!Records) return;
// second step: convert the raw data to the final CONTRACT format
	int i;
	for(i=0; i<Records; i++) {
		CONTRACT* O = dataAppendRow(2,9);
		O->time = dataVar(1,i,0);
		string PC = dataStr(1,i,3);
		string EA = dataStr(1,i,4);
		O->Type = ifelse(*PC == 'P',PUT,CALL) + ifelse(*EA == 'E',EUROPEAN,0);
		int Expiry = dataInt(1,i,1); 
		O->Expiry = 10000*(Expiry%10000) + Expiry/10000; // MMDDYYYY -> YYYYMMDD
		O->fStrike = dataVar(1,i,2);
		O->fAsk = dataVar(1,i,5);
		O->fBid = dataVar(1,i,6);
		O->fVol = dataVar(1,i,7);
		O->fVal = dataVar(1,i,8); // open interest
                O->fUnl = dataVar(1,i,9);
		if(!progress(100*i/Records,0)) break; // show a progress bar
	}
	dataSort(2);
	dataSave(2,strf("History\\%s_%1.t8",Ticker,Year));
}

Last edited by interzonez; 12/21/19 05:55.
Re: iVolatility options csv file to .t8 Zorro file [Re: SBGuy] #478800
12/21/19 06:00
12/21/19 06:00
Joined: Dec 2019
Posts: 6
I
interzonez Offline
Newbie
interzonez  Offline
Newbie
I

Joined: Dec 2019
Posts: 6
Update - I changed the output code to hard code the export file name:

dataSave(2,strf("History\\SPX_2019.t8",Ticker,Year));

I now have the .t8 file though can't interrogate it with ZHistoryEditor

Will try further tests

Re: iVolatility options csv file to .t8 Zorro file [Re: SBGuy] #478806
12/21/19 17:14
12/21/19 17:14
Joined: May 2018
Posts: 134
S
SBGuy Offline OP
Member
SBGuy  Offline OP
Member
S

Joined: May 2018
Posts: 134
A few problem areas I can see right off the bat.

1. Your source data doesn't looke like a CSV. I don't see any commas.
2. Your Expiry is 12/20/2019. That's not an Integer expected by format "i"
3. Your 4th element in your data is supposed to be Strike. I see "C"

Date Expiry Under Strike Bid Ask OI Imp. vol fwd price Type
05/28/2019 12/20/2019 SPX C 100 2672.20 2675.80 533 1.696005 2810.23 E

So just go through the source data very carefully and match it up with the Format string.
You might need another step to pre-process your source file to match the Zorro Format string. Getting data into Zorro t8 file is a process, not automatic. The sample script is just a guide.

Or... get another data source with better initial data format.

Page 1 of 2 1 2

Moderated by  Petra 

Gamestudio download | chip programmers | Zorro platform | shop | Data Protection Policy

oP group Germany GmbH | Birkenstr. 25-27 | 63549 Ronneburg / Germany | info (at) opgroup.de

Powered by UBB.threads™ PHP Forum Software 7.7.1