print(TO_CSV,...) too slow?

Posted By: Sinuhet

print(TO_CSV,...) too slow? - 01/05/19 00:25

Hi all
I would like to export some data as recommended by JCL here.
The export of 1min 1 year 1 forex pair OHLC data takes 2 to 3 minutes depending which from the below codes I use.
This is the time needed to write the CSV file with some 358636 rows (1 min data of EURUSD for 2013) and 6 columns.
This AFTER I turned off the Windows Defender on Windows 10 (otherwise it takes about an hour).

Can I optimise the speed of print(TO_CSV,...) ?
I am asking since the same task takes ZHistoryEditor about 8 sec.
Also the Multicharts are significantly faster in the CSV export.
Thx for help

Code 1 (2 mins):

Code:

function run()
{
  BarPeriod = 1;		  
  StartDate = 2013;
  EndDate = 2013;		
  LookBack = 0;

  print(TO_CSV,"%02i/%02i/%02i %02i:%02i, %.5f, %.5f, %.5f, %.5fn",
		  day(),month(),year(),hour(),minute(),
		  priceOpen(),priceHigh(),priceLow(),priceClose());
}

Code 2 (3 mins):

Code:

function run()
{
  BarPeriod = 1;		
  StartDate = 2013;
  EndDate = 2013;		
  LookBack = 0;

  string line = strf(
    "%02i/%02i/%02i %02i:%02i, %.5f, %.5f, %.5f, %.5fn",
    day(),month(),year()%100,hour(),minute(),
    priceOpen(),priceHigh(),priceLow(),priceClose());
  
  if(is(INITRUN))
  file_delete("Dataexport.csv");
  else
  file_append("Dataexport.csv",line);
}

Posted By: Spirit

Re: print(TO_CSV,...) too slow? - 01/05/19 07:23

Dont print to csv, print to a large string and at the end save the string to a csv file.

Posted By: Sinuhet

Re: print(TO_CSV,...) too slow? - 01/09/19 12:28

Can someone please advise me what function in this 2 step process I should use?

sprintf or strf for printing into string?

and then at the end of run()

if(is(EXITRUN))
print(TO_CSV,...); ?

I am asking since not sure how big string can be optimally used here. Remember 1min OHLC data have some 370000 rows with 5-6 columns. Most of them containing 5 decimal numbers.

Any guidance or modifying the above code would be highly appreciated.
It is all about how to create a CSV file in zorro in the fastest way.

Posted By: Spirit

Re: print(TO_CSV,...) too slow? - 01/09/19 18:49

I think its 3 steps, first strf, then copying the string to a large byte array and in the exitrun, save the array to a file.

Its the file saving that is slow.

Posted By: Sinuhet

Re: print(TO_CSV,...) too slow? - 01/09/19 19:56

Thank you Spirit
indeed that what I have done, but still cannot get the output right (getting a CSV only 1 row of data for the 31.12.2013).
Any idea what is wrong with my code?

Code:

function run()
{
  BarPeriod = 1;		
  StartDate = 20130101;
  EndDate = 20131231;		
  LookBack = 0;

  string line = strf("%02i/%02i/%02i %02i:%02i, %.5f, %.5f, %.5f, %.5fn",
    day(),month(),year()%100,hour(),minute(),
    priceOpen(),priceHigh(),priceLow(),priceClose());
  
  if(is(INITRUN))
  file_delete("Data\\export.csv");
  else
  char str[50000];
  strcat(str,line);
  
  if(is(EXITRUN))
  file_write("Data\\export.csv",str,0);
 }

Posted By: jcl

Re: print(TO_CSV,...) too slow? - 01/10/19 19:20

Your else clause is missing the brackets. Also, allocate the array with malloc in the INITRUN, and check before the strcat call that its length is not exceeded.

Posted By: Sinuhet

Re: print(TO_CSV,...) too slow? - 01/14/19 13:08

Thank you Johann
I have to admit I have minimal experience with malloc, so before I will give a go, may I ask you what would be your expected script time? The time to write a csv with some 380.000 rows and 5 columns was 2-3 min with spripts using either file_append or printtocsv (see above).
Thank you
Sinuhet

Posted By: Zheka

Re: print(TO_CSV,...) too slow? - 01/14/19 22:02

The simplest way would be:

dataLoad(..)
dataSaveCSV(..)

How long would it take?