Hello all,
after Dusktrader's excellent trading script, I wa eager to create some other strategy's. Come to find out, I don't have access to the history that I need from my trading account with Finfx. I don't have a fxcm account so I went to histdata and downloaded a sample of their data.
the data is formatted as follows:
Row Fields:
DateTime Stamp,Bid Quote,Ask Quote,Volume

DateTime Stamp Format:
YYYYMMDD HHMMSSNNN

Legend:
YYYY – Year
MM – Month (01 to 12)
DD – Day of the Month
HH – Hour of the day (in 24h format)
MM – Minute
SS – Second

I've attached my script which doesn't seem to be able to move the open,close,high and low to the TICK structure and doesn't seem to create the .bar file. It also aborts after the third time thru.
I'm out of ideas. Would appreciate any help any one can bring.
Thanks
P

//convert History downloaded from www.histdata.com
//download each year, change in excel to use a comma delimited file, if not already
//sorted so the latest entry in first as required by zorro.
//This script will read the file defined in the script, convert the records to Zorro's
//format and then write them to a binary file name export.bar to be used by Zorro.
//youo can then rename the file to the Zorro .bar format.
//the format of the histdata.com file is DateTime,Open,High,Low,Close,Volume
//the format for zorron is the tick stru included in include\trading.h
//including OLE Date.counting days since midnight 30 December 1899
//typedef struct TICK{ float fOpen, fClose; float fHigh, fLow; DATE time; // time of the tick in UTC time, OLE date/time format} TICK;
//
//the timezone for the histdata.com files is EST without daylight savings time
//so the conversion must take that into account
#include <default.c>
//#include <litec.h>
#include <stdio.h>
#include <trading.h>

TICK mytick;
int count=0;

typedef struct _SYSTEMTIME {
WORD wYear;
WORD wMonth;
WORD wDayOfWeek;
WORD wDay;
WORD wHour;
WORD wMinute;
WORD wSecond;
WORD wMilliseconds;
} SYSTEMTIME;

API(SystemTimeToVariantTime,oleaut32)
int _stdcall SystemTimeToVariantTime(SYSTEMTIME* lpSystemTime, double* pvtime);

DATE ConvertTime(int Year,int Month,int Day,int Hour,int Minute)
{
SYSTEMTIME Time;
memset(&Time,0,sizeof(SYSTEMTIME));
Time.wYear = Year;
Time.wMonth = Month;
Time.wDay = Day;
Time.wHour = Hour;
Time.wMinute = Minute;
DATE vTime;
SystemTimeToVariantTime(&Time,&vTime);
return vTime;
}

function convertdata(string csv)
{
string nextline = strstr(csv,"\n");//this is the file without the first line
string separator = ",";

int Year,Month,Day,Hour,Minute;
int mysec;
//use the csv string to seperate the date time
//format of datetime is "YYYYMMDD HHMMSS" SS is always zero and is not used
sscanf(strtok(csv,separator),"%4i%2i%2i %2i%2i%2i",&Year,&Month,&Day,&Hour,&Minute,&mysec);
mytick.time = ConvertTime(Year,Month,Day,Hour,Minute);

//separate the other vars last column is vol it is always 0 and is ignored
sscanf(strtok(0,separator),"%f",&mytick.fOpen);
sscanf(strtok(0,separator),"%f",&mytick.fHigh);
sscanf(strtok(0,separator),"%f",&mytick.fLow);
sscanf(strtok(0,separator),"%f",&mytick.fClose);
printf("\n %f %f %f %f %f",mytick.time,mytick.fOpen,mytick.fHigh,mytick.fLow,mytick.fClose);

return nextline;
}

function main()
{
string Name = "History\\AUDCAD_M1_2012.csv"; // name of the CSV file containing M1 data
if(!file_date(Name)){
quit("File not found!");
}
//open and create export.bar
FILE* pfile;
pfile=fopen("export.bar","w+b");
if(pfile==NULL){
quit("\n unable to open bar file");
}

printf("\n%s found export file opened",Name);
string mycontent = file_content(Name);
printf("\n firstline %s",mycontent);
while(mycontent) {
mycontent=convertdata(mycontent);
printf("\n nextline %s", mycontent);
fwrite(&mytick,sizeof(mytick),sizeof(mytick),pfile);
if(ferror(pfile)){
printf("\nfile write error");
}
x++;
}//endwhile
printf("\n before close file");
fclose(pfile);
}

Rather than go through all that, it would be easier to just open an FXCM demo account and download the history that way. Have you tried this yet? FXCM has several pairs and using the Download.c script it's pretty easy to get the ones you want.

Thanks for this. Been really helpful dusktrader. And thanks for asking this Pork.

Still, this script is quite interesting because Pork did some important homework there, especially with dateformat munging, parsing etc... Previously when I needed to convert date formats, I was too lazy to discover how to do it properly, so I cheated.

I will take a peek Pork, and see if the script can be finished. Thanks for your work so far.

Pork, you're describing their tick data in the beginning of your post, yet it seems that you eventually downloaded their M1 data. I will assume that you wanted to import the latter, that should be much easier to accomplish. But later I will build upon that, to create bar synthesizer, so that we can also import tick data and convert it to M1, and some other interesting formats. At least, that's the idea, currently all those windows calls and data structures look very spooky to me.

Here's what my sample import data looks like:



Downloaded from http://www.histdata.com/download-free-fo.../eurusd/2013/12

Thanks Dusk trader, I did not know about the FXCM demos. will give that a try.
@acidburn.. I beleive I loaded the M1 data. Here's a sample:
20120102 020200,1.04143,1.04152,1.04143,1.04152,0
20120102 023700,1.04072,1.04075,1.0395,1.04075,0
20120102 023800,1.0412,1.0412,1.04069,1.04088,0
20120102 023900,1.04085,1.04095,1.04059,1.04095,0
20120102 024000,1.04093,1.04097,1.04073,1.04094,0
20120102 024100,1.04092,1.04103,1.0409,1.041,0
20120102 024200,1.04087,1.04108,1.04087,1.04108,0
20120102 024300,1.04105,1.04108,1.04094,1.04097,0
20120102 024400,1.04098,1.04108,1.04098,1.04102,0

Hopefully you have better luck than I did finding the issue or issues. I've tried a number of things over the last few days without effect. I appreciate your efforts.
P

Lots of stuff is confusing, yes. Zorro crashes left and right. It crashes so often that I had to move the icon closer, so I can restart it faster.

Definitely hard working without the debugger...

The data I downloaded uses ';' as a separator, but otherwise looks the same. A different separator should not be a big issue.

And here's the first version for the public. It's small, simple, beautiful and it actually works. Unfortunately, it's still not doing what Zorro expects, because bar files are backwards in time. &^%^$$#@#&*!

I'm afraid coding that last requirement to make bar file fully Zorro compliant will make it much uglier, thus I decided to post this intermediate version. I still think that bar files are poorly designed, CPU cycles are cheap, human time is precious.

While I'm adding the final piece of magic, if anybody can quickly explain how to add a new instrument to Zorro would be helpful. Or point to an existing forum post, I'm sure it popped up before and I might have it bookmarked, but I'll never find it in dozens of bookmarks I collected. This will come handy to not overwrite existing history files, but test with new instrument names.

WOW! Truly amazing!
My idea for the zorro requirement of reverse order was to sort the csv file in excel before running the script. Otherwise a bubble sort should work. tho not very effecient.
as far as the bar file goes I was just going to rename.
Thanks acidburn.
Adding a new instrument is explained on the data import page. There is also a slightly confusing sript to create the needed entries.
P

Done! The generated export.bar file now has bars in reverse order, as required by Zorro. Instead of slowly parsing csv backwards, or stuffing all bars in the memory to write them out in reverse at the very end, I decided to go with a temporary file. It should be the most efficient approach of the three (having in mind that Zorro is an ancient 32bit architecture). Will pay off later when I choke Zorro with gigabytes of tick data.

The script has not been tested, I leave that to you. If you find any bugs, do report!