It is stated numerous times all over the manual that
Quote:
Historical price data for backtests can not be downloaded from IB
or that they are very slow and limited.
This proves true - with Zorro. But not with other platforms. I have been using both Multicharts and Sierra with IB, and historical data download has always happened quite swiftly.
Just did a test, downloading 5 days of EUR.JPY 1 min data. MultiCharts- <1min Zorro - 8min - stopped after loading 1 day..
Looking at the IB Gateway logs, the reasons are as follows:
1. Zorro requests data in 36000s(=10hrs) chunks, and re-requsts every 5hrs (??!)
Historical Data requests need to be assembled in such a way that only a few thousand bars are returned at a time.
And so, it takes MC 15sec from the request to closing the historical data socket.
2. besides "Ask", Zorro also requests data for "Trades" - which do not exist for FX.
Quote:
2017-10-06 09:20:47.152 [IL] INFO [JTS-EServerSocket-3406] - [1:63:76:1:0:0:0:DET] ReqHistoricalData(20)::[version=6,ID=5,action=null,reqDesc=Symbol=EUR Type=CASH Expiry=null Strike=0.0 Put/Call=? Exchange=IDEALPRO CompExch=null Currency=JPY Multiplier=null IbLocalSymbol=null IbTradingClass=null SecIdType=null SecId=null includeExpired=false needLeadFutureMonth=false needContinuousLeadFutureOnly=false newsSource=null Legs=null Special Info=null,endDateTimeStr=20171006 13:20:46 GMT,backfillDuration=36000 S, whatToShow=0, barSizeSettingStr=1 min,formatDate=2,combo=null] 2017-10-06 09:20:47.152 [IL] INFO [JTS-EServerSocket-3406] - [1:63:76:1:0:0:0:DET] [6;5;0;EUR;CASH;null;0;2;null;IDEALPRO;JPY;null;null;false;false;20171006 13:20:46 GMT;1 min;36000 S;0;TRADES;false] ....... 2017-10-06 09:20:47.152 [IL] INFO [JTS-cashhmdsDispatcherS42-3413S42-3414] - cdebug: QUERY | WARNING | qm@305bef71 | Query error | 1755;;EUR.JPY@IDEALPRO Trades;;1;;true;;0;;I | No historical market data for EUR/CASH@FXSUBPIP Last 60
It should have been "Bid" (whattoshow=3), correct?
3. There is a huge lot of strange UnSet() messages generating errors.
Quote:
2017-10-06 09:20:49.031 [IL] INFO [JTS-EServerSocket-3406] - [1:63:76:1:0:0:0:INFO] Handling incoming UnSet(0) message. 2017-10-06 09:20:49.031 [IL] INFO [JTS-EServerSocket-3406] - [1:63:76:1:0:0:0:ERR] Invalid incoming request type - 0
There are also numerous ReqMrktData calls in the first 27 seconds of connection and ReqAccData in the very beginning (even though I pressed [Test], not [Trade]. Script is only "UpdateDays=5"). The first call for historical data is 27sec after pressing the "Test" button (2sec for MC).
Log files are attached.
Btw: MC opens 2 client connections in IB Gateway
Last edited by Zheka; 10/06/1718:26.
Re: IB Plugin needs a fix
[Re: Zheka]
#468489 10/07/1706:3110/07/1706:31
I do not know of a Zorro problem with IB historical data. Nor that Zorro requests "5 hours" or "trades". Only prices and volume are requested, and only at session start. If you cannot download some specific IB data, please contact Support.
IB has strongly limited their historical data until 2016. That was mentioned in the manual, maybe that has confused you. The current limits are listed on the IB website. It's normally 1 year. Zorro, and any other software, can download all historical data within those limits.
Re: IB Plugin needs a fix
[Re: jcl]
#468492 10/07/1714:3610/07/1714:36
It would help immensely if instead of a blanket "there is no problem", we could have a fact-based, productive conversation.
I did not manually create those logs. Run a simple script:
Quote:
function run() { UpdateDays=5; quit; }
set logging level in IBGateway to "Detail" and see for yourself how exactly the api requests are formed, how often and which ones generate errors.
I really want to make it work and your any support would be greatly appreciated. Please post your log.
P.S: - on "5hrs": I underlined the relevant fields in the quoted API requests above - historical data availability from IB was not the point of my post.
Re: IB Plugin needs a fix
[Re: Zheka]
#468507 10/08/1719:2210/08/1719:22
Is the IB plugin invoking a data request for every BrokerHistory2 call? I did the same thing with the Ally plugin because it was easier to implement it that way than to make my own tick buffer.
BrokerHistory2 has a 300-tick limit. Can we perhaps increase the limit or otherwise implement a BrokerHistory3 function where the plugin can return the number of ticks (say, 100,000 ticks) and point to a static/global array of ticks in the plugin as an output?
Yes, it does look like a (legacy?) 300-tick limitation
Quote:
nTicks Input, maximum number of ticks to be filled; guaranteed to be 300 or less. ticks Output, array of up to 300 T6 structs (defined in includetrading.h) to be filled with the ask price history
in filling-in t6 structures internally by Zorro is the reason for current IB plugin re-requesting data for each 5-hr step back in history.
Lifting this limit (and correcting plugins) would speed up loading historical data significantly.
Re: IB Plugin needs a fix
[Re: Zheka]
#468512 10/09/1701:2910/09/1701:29
Yes, all newer plugins use BrokerHistory2. The maximum number of ticks per call is 300, but it can be less dependent on the time period. This has no effect on speed.
- Zheka: Please read my response. I never say that there is no problem, only that I do not know your problem, and you should therefore contact Support. If you have an issue that the other users don't have, we must find out what's different - otherwise we cannot help. It's as simple as that. Your code produces no log, it just sets a variable and quits, so I do not see the problem with that. Downloading data from the broker is described in the manual. You can use the Download script or a simple function:
And if this does not work for you, just explain the problem to Support, and submit as many logs as you want. They can help only when they know the problem, and can reproduce it.
Re: IB Plugin needs a fix
[Re: jcl]
#468526 10/09/1710:0710/09/1710:07
It is obviously less efficient than requesting 1 day of data in 1 go, AND it leads to "pacing violations" on IB API much faster.
Quote:
Your code produces no log, it just sets a variable and quits
I have all the time referred to API messages logged by IB Gateway.
Quote:
I never say that there is no problem, only that I do not know your problem, and you should therefore contact Support
OK, will do. To be frank, I thought support reads this forum and there is no need to do the same thing twice.
Quote:
If you have an issue that the other users don't have
1. It's just that not many users know what's normal and/or know where to look and/or dare to ask. 2. It is all over the manual that getting IB historical data is slow and should not be relied upon. This really puzzled me, since for the last 4 years I have been freely downloading historical data in MC quite swiftly. IB's "1 year" limitation has been more of a 'disclaimer'; downloading 6-7 years of 1min data in one go has never been a problem. Yes, with occasional 'pacing violations' but all in all user experience had been smooth. So, I know it works, and so Zorro should be able to do it similarly well.
Re: IB Plugin needs a fix
[Re: Zheka]
#468527 10/09/1710:1710/09/1710:17
You can easily see with the script above that the 1 year limitation and the slowness really exist. And it has nothing to do with 300 ticks. With 3000 ticks it would be not faster. When you claim you can download more years or faster with other software, then my first suspect is that you're simply downloading it in lower resolution.
Since it makes obviously not much sense to speculate here - if you think you have a technical issue, get to support. They do not read this forum. If it's a bug, help is free. And suggestions are free anyway.
Re: IB Plugin needs a fix
[Re: jcl]
#468529 10/09/1710:2310/09/1710:23