3 registered members (AndrewAMD, VoroneTZ, dpn),
1,346
guests, and 10
spiders. |
Key:
Admin,
Global Mod,
Mod
|
|
|
Easiest way to get 1m History?
#483875
08/08/21 10:56
08/08/21 10:56
|
Joined: Aug 2021
Posts: 237
Lapsa
OP
Member
|
OP
Member
Joined: Aug 2021
Posts: 237
|
I like Zorro's backtesting speed and parameter optimization. Would like to utilize that.
I would like to feed it whole MATICUSDT price history.
How do I do that? It fails to download it from Binance via Download script (did log in successfully).
From the help file - found a script example that can pull ohlc`s from coinapi and wrote:
``` function run() { set(LOGFILE); BarPeriod = 1; asset("MATICUSDT"); vars Prices = series(price()); // Download Bitcoin OHLC prices from Coin.io int Month, Year = 2021, MinutesPerMonth = 31*1440; for(Month = 1; Month <= 12; Month++) { string URL = strf("https://rest.coinapi.io/v1/quotes/BINANCE_SPOT_MATIC_USDT/history?period_id=1MIN&time_start=%04d-%02d-01&limit=%d&apikey=123-API-KEY", Year,Month,MinutesPerMonth);
string Name = strf("MATICUSDT_%04d.t6",Year); assetSource(Name,URL, "X-CoinAPI-Key: 123-API-KEY",0, "[,time_period_end,%Y-%m-%dT%H:%M:%SZ,price_high,price_low,price_open,price_close,,volume_traded"); assetHistory(0,FROM_SOURCE); }
} ```
It does show it in browser but for some reason - Zorro fails to pick it up. Maybe I already exhausted request quota - not sure.
Why it's such a hassle?
|
|
|
Re: Easiest way to get 1m History?
[Re: AndrewAMD]
#483877
08/09/21 09:05
08/09/21 09:05
|
Joined: Aug 2021
Posts: 237
Lapsa
OP
Member
|
OP
Member
Joined: Aug 2021
Posts: 237
|
Glad to see an answer. Means forum is alive.
Got this line in AssetsFix.txt file:
MATIC/USDT,0.730530,0.000150000,0.1665,-0.4580,0.00010000,0.10000,7.305,0,1000,0.0,MATICUSDT
As I understand - the really important bit is the last item that identifies symbol.
If I change it to anything else like this: MATIC/USDT,0.730530,0.000150000,0.1665,-0.4580,0.00010000,0.10000,7.305,0,1000,0.0,MATICUSDT_INVALID
And try to download M1, Zorro says: !BrokerHistory2: Symbol MATICUSDT_INVALID not found
If I change it back and try to download 2020-2021, whole thing runs smoothly but the both t6 files are only 47KB long. They basically are empty.
If I try to download Ticks, it says: Error 063: Tick history not available
--------------------
Tried to do whole thing manually. Downloaded bunch of csv`s through binance api. Did some cleanup so it looks like this:
Date,Open,High,Low,Close,Volume,QuoteVolume 26/04/2019 18:00:00,0.00263000,0.01052000,0.00263000,0.00639000,157258876.10000000,1092100.13082700
Modified CSVtoHistory script format to: string Format = "%d/%m/%Y %H:%M:%S,f3,f1,f2,f4,f6";
And ran import. Seemingly successfully.
Ran CSVfromHistory to see what has made in and it looks decent: 2021-01-01T00:00:00,0.01742,0.01742,0.01742,0.01742,0.00000,5312.29980 2021-01-01T00:01:00,0.01741,0.01741,0.01740,0.01740,0.00000,79141.60156 2021-01-01T00:02:00,0.01739,0.01739,0.01735,0.01735,0.00000,157942.59375 2021-01-01T00:03:00,0.01736,0.01736,0.01735,0.01735,0.00000,46928.89844
But when I try to run Ehler's AMFM script or Workshop4 or Workshop5, it just says: Error 047: No bars generated
Sometimes: Error 042: no MATICUSDT parameters at walk 0 ^ tried to change MATIC/USDT in assets list to MATICUSDT for the name too
And Zorro Chart Viewer says: Unspecified error
Tried setting various StartDate and EndDate. Nothing changed. Tried setting BarPeriod to 1, 5, 60. Nothing changed.
Why does it dislike that imported data?
Also - failed to find the log files. Tried to set LOGFILE flag but that produced nothing.
Last edited by Lapsa; 08/09/21 09:07.
|
|
|
Re: Easiest way to get 1m History?
[Re: Lapsa]
#483878
08/09/21 12:38
08/09/21 12:38
|
Joined: Feb 2017
Posts: 1,731 Chicago
AndrewAMD
Serious User
|
Serious User
Joined: Feb 2017
Posts: 1,731
Chicago
|
It sounds like Binance does not store very much historical data. Audit the t6 files you downloaded for some hints. Next, there are many helpful suggestions on how to handle each particular error code. Start there: https://zorro-trader.com/manual/en/errors.htm (error codes) Next, did you audit your t6 files? Use the History script (or the Z History Editor from the download section). Is the t6 data sorted from newest to oldest (and not the other way around)? Is the file naming convention identical to what is described in the manual? Also - failed to find the log files. Tried to set LOGFILE flag but that produced nothing. You probably put it in the wrong spot. Post your script.
|
|
|
Re: Easiest way to get 1m History?
[Re: AndrewAMD]
#483879
08/09/21 15:51
08/09/21 15:51
|
Joined: Aug 2021
Posts: 237
Lapsa
OP
Member
|
OP
Member
Joined: Aug 2021
Posts: 237
|
> It sounds like Binance does not store very much historical data. Probably untrue. At least for my case. Got data since its inception 29-4-2019. Did visit manual errors section. Didn't find it helpful. > Next, did you audit your t6 files? Not yet. > Is the t6 data sorted from newest to oldest (and not the other way around)? Hoping that this is the culprit. Will test that now. <------------------ > Is the file naming convention identical to what is described in the manual? Think there's no issue with this. > You probably put it in the wrong spot. Post your script. ``` function run() { set(LOGFILE); ``` ----------------------------------------- I guess it's the sorting issue. Will try to reverse it and import again. ----------------------------------------- Noticed there's a Sort function on the history viewer. Sorted, saved. ----------------------------------------- IT'S ALIVE!!!!!!!!!!!!!!!! I'm so excited.... Such a playground. Thank you, Andrew! Much appreciated.
Last edited by Lapsa; 08/09/21 16:05.
|
|
|
Re: Easiest way to get 1m History?
[Re: Lapsa]
#483882
08/09/21 17:49
08/09/21 17:49
|
Joined: Aug 2021
Posts: 237
Lapsa
OP
Member
|
OP
Member
Joined: Aug 2021
Posts: 237
|
Well... I need to thoroughly RTFM. At least now it kind of seems worth it.
One more quick question - going live would still require manual data feeding or Zorro would somehow figure it out and pick up price data via Websocket?
-----------------------------
For the sake of learning, squashed together this script:
``` #include <profile.c>
function run() { BarPeriod = 1; StartDate = 20210729; EndDate = 20210809; Verbose = 2; LookBack = 300; asset("MATICUSDT"); set(LOGFILE|PLOTNOW); set(PARAMETERS);
vars Prices = series(priceClose()); int SrsiPeriod = optimize(5, 5, 60, 1, 2); vars SRSI = series(StochRSI(Prices, SrsiPeriod, 3, 3, MAType_KAMA)); int AtrMultiplier = optimize(0.1, 0.1, 4, 0.1, 1); int AtrPeriod = optimize(3, 3, 100, 1, 1); Stop = AtrMultiplier*ATR(AtrPeriod); if(crossOver(SRSI,30)) enterLong(); else if(crossUnder(SRSI,70)) enterShort(); } ```
Little SRSI cross strat that runs on KAMA just because I could feed it in.
First question would be - how come there's only 1 SRSI param for period? So far I've seen that SRSI runs on 2 periods: 1 for RSI, 1 for stoch.
Second question would be about ridiculous results:
``` Read stoch_rsi.par Test: stoch_rsi MATICUSDT 2021 Warning 035: MATICUSDT outlier at #7021 2021-08-01 1.0057->1.0877 Warning 035: MATICUSDT outlier at #14040 2021-08-08 1.0779->1.1966 Warning 035: MATICUSDT outlier at #14041 2021-08-08 1.1966->1.0872 Monte Carlo Analysis... Median AR 16657% Win 1985$ MI 5492$ DD 225$ Capital 733$ Trades 1308 Win 49.5% Avg +15.2p Bars 10 AR 8989% PF 2.22 SR 0.00 UI 0% R2 1.00
Chart... Copied to Clipboard! ```
What's an outlier? Imported price data is still invalid?
What's up with that annual return? I guess it doesn't take fees into account but still - number looks stupid.
How would you go about debunking that?
Nevermind. Figured out what's happening. All in all - Zorro looks decent. Hopefully live trading doesn't flop.
Time to learn.
Last edited by Lapsa; 08/09/21 20:50.
|
|
|
Re: Easiest way to get 1m History?
[Re: Lapsa]
#484038
09/01/21 08:33
09/01/21 08:33
|
Joined: Feb 2021
Posts: 19
CpOo
Newbie
|
Newbie
Joined: Feb 2021
Posts: 19
|
Hi, Binance has data since inception and you can retrieve them all with Download.c script or with assetHistory function. Problem with Download.c script is that you will need to click Download each 1500 ticks. Thats almost 1 time per day as a day has 1440 ticks. I have a simple script that download the full year and I am happy to share it here. Please note that I am not a coder, and the script hangs at the end... but the data is downloaded. Also note this report from yesterday: https://opserver.de/ubb7/ubbthreads.php?ubb=showflat&Number=484037#Post484037 Using this method also insert 1 minute gap between each 1500 ticks.
function main()
{
StartDate = 2021;
EndDate = 2021;
//NumYears = 1; //
Outlier = 0;
string Name = "MATICUSDT";
while(Name)
assetHistory(Name,1);
quit();
}
That code works well with Binance Futures plugin. I did not test with the other Binance plugin. About your Outlier warning. I use Outlier = 0; (as in the script above) to deactivate the Outlier detection. With crypto is also important to have this line in your run() function because trading is 24/7 non stop:
function run()
{
resf(BarMode, BR_WEEKEND+BR_MARKET);
...
Finally, be very careful with Asset conf in the .csv. It's critical to have the correct values for PIP, PIPCost, and LotAmount. I have this line for MATICUSDT (Binance Futures):
Name,Price,Spread,RollLong,RollShort,PIP,PIPCost,MarginCost,Leverage,LotAmount,Commission,Symbol
MATICUSDT,2.00,0.0001,0,0,0.0001,0.0001,-2,0,1,-0.072,*
You can find the base values at: https://www.binance.com/en/futures/trading-rules but you will need to calculate some of them: LotAmount = "Min. Trade Amount". It's the second column: 1 for MATIC PIP = "Min. Price Movement". It's the second part of the third column (Price in USDT). 0.0001 USDT for MATIC PIPCost: Just multiply LotAmount with PIP: 1*0.0001 = 0.0001 MarginCost: It's de initial margin rate (calculated from leverage) and you can find it here: https://www.binance.com/en/support/faq/360033162192. If you want to work with 50x leverage the initial margin rate is 2%, so MarginCost = -2 in the config file. Hope this helps.
|
|
|
|