Gamestudio Links
Zorro Links
Newest Posts
Blobsculptor tools and objects download here
by NeoDumont. 03/28/24 03:01
Issue with Multi-Core WFO Training
by aliswee. 03/24/24 20:20
Why Zorro supports up to 72 cores?
by Edgar_Herrera. 03/23/24 21:41
Zorro Trader GPT
by TipmyPip. 03/06/24 09:27
VSCode instead of SED
by 3run. 03/01/24 19:06
AUM Magazine
Latest Screens
The Bible Game
A psychological thriller game
SHADOW (2014)
DEAD TASTE
Who's Online Now
2 registered members (AndrewAMD, Nymphodora), 972 guests, and 8 spiders.
Key: Admin, Global Mod, Mod
Newest Members
sakolin, rajesh7827, juergen_wue, NITRO_FOREVER, jack0roses
19043 Registered Users
Previous Thread
Next Thread
Print Thread
Rate Thread
Page 1 of 2 1 2
Easiest way to get 1m History? #483875
08/08/21 10:56
08/08/21 10:56
Joined: Aug 2021
Posts: 237
L
Lapsa Offline OP
Member
Lapsa  Offline OP
Member
L

Joined: Aug 2021
Posts: 237
I like Zorro's backtesting speed and parameter optimization. Would like to utilize that.

I would like to feed it whole MATICUSDT price history.

How do I do that? It fails to download it from Binance via Download script (did log in successfully).

From the help file - found a script example that can pull ohlc`s from coinapi and wrote:

```
function run()
{
set(LOGFILE);
BarPeriod = 1;
asset("MATICUSDT");
vars Prices = series(price());


// Download Bitcoin OHLC prices from Coin.io
int Month, Year = 2021, MinutesPerMonth = 31*1440;
for(Month = 1; Month <= 12; Month++) {
string URL = strf("https://rest.coinapi.io/v1/quotes/BINANCE_SPOT_MATIC_USDT/history?period_id=1MIN&time_start=%04d-%02d-01&limit=%d&apikey=123-API-KEY",
Year,Month,MinutesPerMonth);

string Name = strf("MATICUSDT_%04d.t6",Year);
assetSource(Name,URL,
"X-CoinAPI-Key: 123-API-KEY",0,
"[,time_period_end,%Y-%m-%dT%H:%M:%SZ,price_high,price_low,price_open,price_close,,volume_traded");
assetHistory(0,FROM_SOURCE);
}

}
```

It does show it in browser but for some reason - Zorro fails to pick it up. Maybe I already exhausted request quota - not sure.

Why it's such a hassle?

Re: Easiest way to get 1m History? [Re: Lapsa] #483876
08/08/21 15:53
08/08/21 15:53
Joined: Feb 2017
Posts: 1,718
Chicago
AndrewAMD Online
Serious User
AndrewAMD  Online
Serious User

Joined: Feb 2017
Posts: 1,718
Chicago
The Download script is easiest.

You say downloading via Binance did not work. What went wrong? What is your asset list configuration? What error message did you receive? What did you do to correct an error on your end based on what the error messages told you?

Share your logfile.

https://zorro-project.com/manual/en/account.htm (Asset List)
https://manual.zorro-project.com/binance.htm (Binance plugin)
https://zorro-trader.com/manual/en/trouble.htm (troubleshooting)
https://zorro-trader.com/manual/en/errors.htm (error codes)

Re: Easiest way to get 1m History? [Re: AndrewAMD] #483877
08/09/21 09:05
08/09/21 09:05
Joined: Aug 2021
Posts: 237
L
Lapsa Offline OP
Member
Lapsa  Offline OP
Member
L

Joined: Aug 2021
Posts: 237
Glad to see an answer. Means forum is alive.

Got this line in AssetsFix.txt file:

MATIC/USDT,0.730530,0.000150000,0.1665,-0.4580,0.00010000,0.10000,7.305,0,1000,0.0,MATICUSDT

As I understand - the really important bit is the last item that identifies symbol.

If I change it to anything else like this:
MATIC/USDT,0.730530,0.000150000,0.1665,-0.4580,0.00010000,0.10000,7.305,0,1000,0.0,MATICUSDT_INVALID

And try to download M1, Zorro says:
!BrokerHistory2: Symbol MATICUSDT_INVALID not found

If I change it back and try to download 2020-2021, whole thing runs smoothly but the both t6 files are only 47KB long.
They basically are empty.

If I try to download Ticks, it says:
Error 063: Tick history not available

--------------------

Tried to do whole thing manually.
Downloaded bunch of csv`s through binance api.
Did some cleanup so it looks like this:

Date,Open,High,Low,Close,Volume,QuoteVolume
26/04/2019 18:00:00,0.00263000,0.01052000,0.00263000,0.00639000,157258876.10000000,1092100.13082700

Modified CSVtoHistory script format to:
string Format = "%d/%m/%Y %H:%M:%S,f3,f1,f2,f4,f6";

And ran import. Seemingly successfully.

Ran CSVfromHistory to see what has made in and it looks decent:
2021-01-01T00:00:00,0.01742,0.01742,0.01742,0.01742,0.00000,5312.29980
2021-01-01T00:01:00,0.01741,0.01741,0.01740,0.01740,0.00000,79141.60156
2021-01-01T00:02:00,0.01739,0.01739,0.01735,0.01735,0.00000,157942.59375
2021-01-01T00:03:00,0.01736,0.01736,0.01735,0.01735,0.00000,46928.89844

But when I try to run Ehler's AMFM script or Workshop4 or Workshop5, it just says:
Error 047: No bars generated

Sometimes:
Error 042: no MATICUSDT parameters at walk 0
^ tried to change MATIC/USDT in assets list to MATICUSDT for the name too

And Zorro Chart Viewer says:
Unspecified error

Tried setting various StartDate and EndDate. Nothing changed.
Tried setting BarPeriod to 1, 5, 60. Nothing changed.

Why does it dislike that imported data?

Also - failed to find the log files. Tried to set LOGFILE flag but that produced nothing.

Last edited by Lapsa; 08/09/21 09:07.
Re: Easiest way to get 1m History? [Re: Lapsa] #483878
08/09/21 12:38
08/09/21 12:38
Joined: Feb 2017
Posts: 1,718
Chicago
AndrewAMD Online
Serious User
AndrewAMD  Online
Serious User

Joined: Feb 2017
Posts: 1,718
Chicago
It sounds like Binance does not store very much historical data. Audit the t6 files you downloaded for some hints.

Next, there are many helpful suggestions on how to handle each particular error code. Start there:
https://zorro-trader.com/manual/en/errors.htm (error codes)

Next, did you audit your t6 files? Use the History script (or the Z History Editor from the download section). Is the t6 data sorted from newest to oldest (and not the other way around)? Is the file naming convention identical to what is described in the manual?
Originally Posted by Lapsa
Also - failed to find the log files. Tried to set LOGFILE flag but that produced nothing.
You probably put it in the wrong spot. Post your script.

Re: Easiest way to get 1m History? [Re: AndrewAMD] #483879
08/09/21 15:51
08/09/21 15:51
Joined: Aug 2021
Posts: 237
L
Lapsa Offline OP
Member
Lapsa  Offline OP
Member
L

Joined: Aug 2021
Posts: 237
> It sounds like Binance does not store very much historical data.

Probably untrue. At least for my case. Got data since its inception 29-4-2019.

Did visit manual errors section. Didn't find it helpful.

> Next, did you audit your t6 files?

Not yet.

> Is the t6 data sorted from newest to oldest (and not the other way around)?

Hoping that this is the culprit. Will test that now. <------------------

> Is the file naming convention identical to what is described in the manual?

Think there's no issue with this.

> You probably put it in the wrong spot. Post your script.

```
function run()
{
set(LOGFILE);
```


-----------------------------------------

I guess it's the sorting issue. Will try to reverse it and import again.

[Linked Image]


-----------------------------------------

Noticed there's a Sort function on the history viewer.

Sorted, saved.

-----------------------------------------

IT'S ALIVE!!!!!!!!!!!!!!!!

[Linked Image]

I'm so excited.... Such a playground.

Thank you, Andrew! Much appreciated.

Last edited by Lapsa; 08/09/21 16:05.
Re: Easiest way to get 1m History? [Re: Lapsa] #483881
08/09/21 16:31
08/09/21 16:31
Joined: Feb 2017
Posts: 1,718
Chicago
AndrewAMD Online
Serious User
AndrewAMD  Online
Serious User

Joined: Feb 2017
Posts: 1,718
Chicago
You can also use dataSort on the dataset prior to saving to t6 file.
https://www.zorro-project.com/manual/en/data.htm

Re: Easiest way to get 1m History? [Re: Lapsa] #483882
08/09/21 17:49
08/09/21 17:49
Joined: Aug 2021
Posts: 237
L
Lapsa Offline OP
Member
Lapsa  Offline OP
Member
L

Joined: Aug 2021
Posts: 237
Well... I need to thoroughly RTFM. At least now it kind of seems worth it.

One more quick question - going live would still require manual data feeding or Zorro would somehow figure it out and pick up price data via Websocket?

-----------------------------

For the sake of learning, squashed together this script:

```
#include <profile.c>

function run()
{
BarPeriod = 1;
StartDate = 20210729;
EndDate = 20210809;
Verbose = 2;
LookBack = 300;
asset("MATICUSDT");
set(LOGFILE|PLOTNOW);
set(PARAMETERS);

vars Prices = series(priceClose());

int SrsiPeriod = optimize(5, 5, 60, 1, 2);

vars SRSI = series(StochRSI(Prices, SrsiPeriod, 3, 3, MAType_KAMA));

int AtrMultiplier = optimize(0.1, 0.1, 4, 0.1, 1);
int AtrPeriod = optimize(3, 3, 100, 1, 1);

Stop = AtrMultiplier*ATR(AtrPeriod);

if(crossOver(SRSI,30))
enterLong();
else if(crossUnder(SRSI,70))
enterShort();
}
```

Little SRSI cross strat that runs on KAMA just because I could feed it in.

First question would be - how come there's only 1 SRSI param for period?
So far I've seen that SRSI runs on 2 periods: 1 for RSI, 1 for stoch.

Second question would be about ridiculous results:

```
Read stoch_rsi.par
Test: stoch_rsi MATICUSDT 2021
Warning 035: MATICUSDT outlier at #7021 2021-08-01 1.0057->1.0877
Warning 035: MATICUSDT outlier at #14040 2021-08-08 1.0779->1.1966
Warning 035: MATICUSDT outlier at #14041 2021-08-08 1.1966->1.0872
Monte Carlo Analysis... Median AR 16657%
Win 1985$ MI 5492$ DD 225$ Capital 733$
Trades 1308 Win 49.5% Avg +15.2p Bars 10
AR 8989% PF 2.22 SR 0.00 UI 0% R2 1.00

Chart...
Copied to Clipboard!
```

What's an outlier? Imported price data is still invalid?

What's up with that annual return?
I guess it doesn't take fees into account but still - number looks stupid.

How would you go about debunking that?

Nevermind. Figured out what's happening. All in all - Zorro looks decent.
Hopefully live trading doesn't flop.

Time to learn.

Last edited by Lapsa; 08/09/21 20:50.
Re: Easiest way to get 1m History? [Re: Lapsa] #484038
09/01/21 08:33
09/01/21 08:33
Joined: Feb 2021
Posts: 19
C
CpOo Offline
Newbie
CpOo  Offline
Newbie
C

Joined: Feb 2021
Posts: 19
Hi,

Binance has data since inception and you can retrieve them all with Download.c script or with assetHistory function.

Problem with Download.c script is that you will need to click Download each 1500 ticks. Thats almost 1 time per day as a day has 1440 ticks.

I have a simple script that download the full year and I am happy to share it here. Please note that I am not a coder, and the script hangs at the end... but the data is downloaded.

Also note this report from yesterday: https://opserver.de/ubb7/ubbthreads.php?ubb=showflat&Number=484037#Post484037 Using this method also insert 1 minute gap between each 1500 ticks.

Code
function main()
{
  StartDate = 2021;
  EndDate = 2021;
  //NumYears = 1; //
  Outlier = 0;

  string Name = "MATICUSDT";
  while(Name)
  assetHistory(Name,1);
  quit();
}



That code works well with Binance Futures plugin. I did not test with the other Binance plugin.

About your Outlier warning. I use Outlier = 0; (as in the script above) to deactivate the Outlier detection. With crypto is also important to have this line in your run() function because trading is 24/7 non stop:

Code
function run()
{
resf(BarMode, BR_WEEKEND+BR_MARKET);
...


Finally, be very careful with Asset conf in the .csv. It's critical to have the correct values for PIP, PIPCost, and LotAmount. I have this line for MATICUSDT (Binance Futures):

Code
Name,Price,Spread,RollLong,RollShort,PIP,PIPCost,MarginCost,Leverage,LotAmount,Commission,Symbol
MATICUSDT,2.00,0.0001,0,0,0.0001,0.0001,-2,0,1,-0.072,*


You can find the base values at: https://www.binance.com/en/futures/trading-rules but you will need to calculate some of them:

LotAmount = "Min. Trade Amount". It's the second column: 1 for MATIC
PIP = "Min. Price Movement". It's the second part of the third column (Price in USDT). 0.0001 USDT for MATIC
PIPCost: Just multiply LotAmount with PIP: 1*0.0001 = 0.0001
MarginCost: It's de initial margin rate (calculated from leverage) and you can find it here: https://www.binance.com/en/support/faq/360033162192. If you want to work with 50x leverage the initial margin rate is 2%, so MarginCost = -2 in the config file.

Hope this helps.

Re: Easiest way to get 1m History? [Re: Lapsa] #484136
09/14/21 19:29
09/14/21 19:29
Joined: Aug 2021
Posts: 237
L
Lapsa Offline OP
Member
Lapsa  Offline OP
Member
L

Joined: Aug 2021
Posts: 237
@CpOo works well. afaik the new update fixes gap issue too

Re: Easiest way to get 1m History? [Re: Lapsa] #484384
10/18/21 21:14
10/18/21 21:14
Joined: Sep 2021
Posts: 9
Waikato
T
TimExcellent Offline
Newbie
TimExcellent  Offline
Newbie
T

Joined: Sep 2021
Posts: 9
Waikato
Quick question guys also new to this, but the free data here on the Zorro site is that 1minute SPY options data?:

Attached Files Screenshot 2021-10-18 171413.png
Re: Easiest way to get 1m History? [Re: TimExcellent] #484385
10/18/21 21:35
10/18/21 21:35
Joined: Aug 2017
Posts: 294
Netherlands
G
Grant Offline
Member
Grant  Offline
Member
G

Joined: Aug 2017
Posts: 294
Netherlands
Most prob not since it's only 150 MB in size. AFAIK, you need to pay for intraday options data at cboe.com ot other sources.

Re: Easiest way to get 1m History? [Re: Lapsa] #484386
10/18/21 22:07
10/18/21 22:07
Joined: Sep 2021
Posts: 9
Waikato
T
TimExcellent Offline
Newbie
TimExcellent  Offline
Newbie
T

Joined: Sep 2021
Posts: 9
Waikato
Do you know how to produce synthetic 1 min options data from the 1 min SPY data, and output to the History folder?

Re: Easiest way to get 1m History? [Re: Lapsa] #484387
10/18/21 22:31
10/18/21 22:31
Joined: Aug 2017
Posts: 294
Netherlands
G
Grant Offline
Member
Grant  Offline
Member
G

Joined: Aug 2017
Posts: 294
Netherlands
I don't think it's a smart idea. A small difference between real & theoretic option prices wouldn't be a problem for EOD data, but it could have a big impact on the quality of your strategy in a high frequency scenario. I don't work with these data, so I could be wrong on this.

Anyway, you could create these prices in a spreadsheet, using a Black & Scholes or Cox-Ross-Rubinstein model (more precise), save it as an CSV file and convert it by using the dataParse function (see https://manual.zorro-project.com/history.htm)

Page 1 of 2 1 2

Moderated by  Petra 

Powered by UBB.threads™ PHP Forum Software 7.7.1