We got several complaints today that downloading data from Yahoo does not work anymore. The reason is that Yahoo switched their servers to a new HTTPS protocol last weekend.
We've just uploaded a new beta version, 1.56.4, which downloads Yahoo data with the new protocol. You can get it from http://opserver.de/down/ZorrroBeta.zip.
Alternatively, you can still use your old version, but download Yahoo data from Quandl, f.i. assetHistory("YAHOO/AAPL",FROM_QUANDL); You must register at Quandl for getting a Quandl key. Also, Quandl does support Stocks, but not all ETFs in their Yahoo database.
Update: For downloading all the Z8 assets with the beta version, you can use a small script like this:
I tested the Heatmap script built-in Zorro. I tried both V1.56 and V1.58, both of them carshed like below:
Heatmap compiling.............. Bad date format - can't parse Historyhistory.csv 0 AEIS bars read 0 AMAT bars read More assets.. 0 AMD bars read Time 00:01:06 Error 111: Crash in script Time 00:01:06
What did I do wrong?
By the way, does the data from Yahoo have volume information? What's the right function to get the volume info.
Thank you
Jeff
Re: Yahoo has changed their protocol
[Re: Jeff1228]
#465901 05/17/1708:1505/17/1708:15
Yes, Yahoo data has volume info. Or had, at least - look into the "history.csv" with a HTML browser and you'll get the answer for the new download problems.
Re: Yahoo has changed their protocol
[Re: jcl]
#465905 05/17/1709:0705/17/1709:07
Ok, we found that you can download the same ETF data from Google. But I don't know how long, their API is also depreciated. For the moment, it works. So when you want to test or run Z8, run this script before:
I have account and data subscription with IB, and I run your scripts Heatmap and MVO, can I simple change "assetHistory(Names[N],FROM_YAHOO)" to "assetHistory(Names[N],FROM_IB)"? Would it work?
Is your code a seperate script I should run before I run other scripts? If so, should I run it every time before I run others? or I should put it somewhere in your Heatmap and MVO script?
Can I get stocks data from Google?
Last edited by Jeff1228; 05/19/1704:17.
Re: Yahoo has changed their protocol
[Re: Jeff1228]
#465971 05/19/1704:2805/19/1704:28
Dowloading from Google works for me with the default Z8 assets, but maybe not with old Zorro versions or when you have added other assets. If Google does not support the asset, the parse function returns 0 and the script crashes. Printing an error message instead is left as an exercise to the reader.
Google is no final solution since they will also discontinue the API some day. I'm afraid the final solution will possibly not be free.
-The Google download can be run before any script with Yahoo download. The script will detect that the data is already up to date. Google has also stocks, but major stocks are no problem anyway since you get them free from Quandl. The problem are ETFs.
Re: Yahoo has changed their protocol
[Re: jcl]
#465976 05/19/1707:2705/19/1707:27
While the google download script now indeed no longer causes a crash, there is a large discrepancy in the test results of the Z8 system with the google data. The drawdown is more substantial (The previous test was made with the 1.58.0 beta version, whose result I fortunately still had in the log directory), AR was about 18% with leverage 2 and now it is down to 13%. Is googles data adjusted for dividends, splits etc.?
Re: Yahoo has changed their protocol
[Re: hatten]
#465998 05/19/1712:3005/19/1712:30
Only for splits. AFAIK dividends are not included. This affects the result substantially. Also the Google data has a lot more outliers and gaps than the Yahoo data. So this is only a temporary solution.
We will implement a final solution in the next Zorro update.
Re: Yahoo has changed their protocol
[Re: jcl]
#466008 05/19/1719:3905/19/1719:39
If I read their webpage correctly eoddata delivers historical end-of-the-day data for a very affordable one time payment.
Moreover it says: "If you purchase access to 5 years historical end of day quote data then you can download all 5 years and in 3 years time there will be 8 years for you to download. Once you have purchased access, you can download your historical end of day data as many times as you like. If your PC ever crashes and you loose your data simply download it again."
So as time progress you get also access to new data.
Re: Yahoo has changed their protocol
[Re: hatten]
#466011 05/20/1700:5805/20/1700:58
For anyone using the quantmod library in R, Joshua Ulrich (the package's author) released a workaround that you can get by updating quantmod to the development version. In R, do either:
remotes::install_github("joshuaulrich/quantmod", ref="157_yahoo_502") or devtools::install_github("joshuaulrich/quantmod", ref="157_yahoo_502")
I search for a Yahoo API substitute also for another reason than Zorro and I stumbled upon a site, I don't know really what to think of it. alphavantage.co offers free (you can apply a free API key) intraday, daily, weekly and monthly prices for US stock, ETFs etc. Their history e.g. of TLT goes back to 2002, of SPY back to 2001. The output format is JSON but this is to handle I guess. I did not verify the quality of their prices so far. They declare their aim as "democratizing access to institution-grade financial analytics" but they have no address or company name on their site and even the domain name alphavantage.co is registered by an agent it seems. The IP address 216.239.36.21 of their server belongs to a Google net block. As said, I don't know how reputable they are.
Re: Yahoo has changed their protocol
[Re: Sphin]
#466049 05/23/1703:2905/23/1703:29
I use the php solution in my website and it's also working for now. But no one knows if and when Y will tighten its security's features. The cookie's validity of 1 year is e.g. very convenient ...
Re: Yahoo has changed their protocol
[Re: Sphin]
#466127 05/26/1706:4005/26/1706:40
Another way is to use the .NET webbrowser control in order to load the whole website from yahoo, boerse-frankfurt etc. Then we get a HTML document object and can walk through the tables and it's rows and cells to get the quotes. This is working for me, too.
Re: Yahoo has changed their protocol
[Re: Jens]
#466155 05/27/1717:0805/27/1717:08
For many ETFs I get historical prices only for 2 years via the described Yahoo interface. For the same ETFs I get 5 years or more at boerse-frankfurt.de.
Re: Yahoo has changed their protocol
[Re: jcl]
#466157 05/27/1723:3905/27/1723:39
I made this .Net exe from previous links to C# code source code attached.
Put the exe in zorro folder. Then call it with this in Zorro
var dataFromYahoo(string Code, int mode) { string Format; if (mode == FROM_YAHOO | UNADJUSTED) Format = "%Y-%m-%d,f3,f1,f2,,f4,f6";//UnadjAdj Close else if (mode == FROM_YAHOO) Format = "%Y-%m-%d,f3,f1,f2,f4,,f6";//Adj Close
exec("YahooFinanceAPI.exe", Code, 1); //exec("YahooFinanceAPI.exe", "AAPL 1",1); dataNew(1, 0, 7); int numRecord = dataParse(1, Format, "History\history.csv"); printf("n%s %d records read from YAHOO", Code, numRecord); dataSave(1, strf("%s.t6", Code)); return numRecord; }
Re: Yahoo has changed their protocol
[Re: vinsom]
#466214 06/01/1709:0906/01/1709:09
I made this .Net exe from previous links to C# code source code attached.
Put the exe in zorro folder. Then call it with this in Zorro
var dataFromYahoo(string Code, int mode) { string Format; if (mode == FROM_YAHOO | UNADJUSTED) Format = "%Y-%m-%d,f3,f1,f2,,f4,f6";//UnadjAdj Close else if (mode == FROM_YAHOO) Format = "%Y-%m-%d,f3,f1,f2,f4,,f6";//Adj Close
exec("YahooFinanceAPI.exe", Code, 1); //exec("YahooFinanceAPI.exe", "AAPL 1",1); dataNew(1, 0, 7); int numRecord = dataParse(1, Format, "Historyhistory.csv"); printf("n%s %d records read from YAHOO", Code, numRecord); dataSave(1, strf("%s.t6", Code)); return numRecord; }
Thank you but I don't quite understand this.
Where is the exe code?
What's the purpose of the code here? Should I put it into the script hwere I use FROM_YAHOO?
What's the purpose of the attachment?
I am new here but I really need this working.
Jeff
Re: Yahoo has changed their protocol
[Re: Jeff1228]
#466225 06/01/1721:0606/01/1721:06
You need to compile the attached code with MS Visual studio (can't attach exe). This will generate the exe YahooFinanceAPI.exe. Put this in the Zorro folder. Then get the history with the zorro code snippet above, that will call the exe generated, with exec("YahooFinanceAPI.exe", Code, 1);. Cheers
Re: Yahoo has changed their protocol
[Re: vinsom]
#466237 06/02/1708:0506/02/1708:05
Sorry I dont have MS Visual Studio. Can you please upload the exe to wetransfer.com or similar site? Thanks! Update: i tried to build with msbuild but didnt compile because of error.
Last edited by konorti; 06/02/1719:30.
Re: Yahoo has changed their protocol
[Re: konorti]
#466247 06/02/1723:5406/02/1723:54
Can I use the google script provided by jcl to download Canadian stocks history from Yahoo, I tried but got this
google compiling..............
Bad - can't parse Historyhistory.csv Time 00:01:14
Last time I used an assetList with all US stocks, it worked.
I copied the link "https://www.google.com/finance/historical?q=AC.TO&startdate=01-Jan-2000" into the browser, actually it worked and I got the data up to date.
What's wrong here?
Thank you
Jeff
Last edited by Jeff1228; 06/07/1711:10.
Re: Yahoo has changed their protocol
[Re: konorti]
#466305 06/07/1708:3706/07/1708:37
Thanks! I also modified the "Download" strategy to use your exe.
Hi,
I am using MVO script and an assetlist with all Canadian stocks. I tried your method to download data from YAHOO. I put the .exe to the Zorro fold and your "var dataFromYahoo(string Code, int mode)" snippet as a function into the MVO script ahead of run function, changed orignal "assetHistory(Names[N],FROM_YAHOO);" to "dataFromYahoo(Names[N],FROM_YAHOO);". When I run it, for each of the assets, it seemed that a blank window pooped up waiting for some input, but I could not type in anything. When I crossed it out, Zorro seemed to download data. But at the end, the data is all old, not updated!
What's wrong with my script?
Thank you
Last edited by Jeff1228; 06/07/1709:11.
Re: Yahoo has changed their protocol
[Re: vinsom]
#466319 06/08/1708:4106/08/1708:41
I think pop-up empty window is fine. This works the same as original Download script that you have to write asset name and press "Load Yahoo Data" then. Try the the code snipet from Download_Y script as I modified a little. Naming of the CSV file didnt work for me in the original script, so I changed it.
Zorro should print something like this:
Download_Y compiling........... FXCM.dll => FXCM plugin mounted Login FXCM.. at UTC 06-08 12:10 Downloading TLT.. Parse Historyhistory.csv.. 2017-06-07,125.239998,125.519997,124.790001,124.919998,124.919998,11951200 42893.00000000 125.2400 125.5200 124.7900 124.9200 11951200.0000 . 2017-06-06,125.690002,125.870003,125.400002,125.519997,125.519997,12841500 42892.00000000 125.6900 125.8700 125.4000 125.5200 12841500.0000 nTLT 3741 records read from YAHOO
Last edited by konorti; 06/08/1712:20.
Re: Yahoo has changed their protocol
[Re: konorti]
#466329 06/09/1702:2506/09/1702:25
I think pop-up empty window is fine. This works the same as original Download script that you have to write asset name and press "Load Yahoo Data" then. Try the the code snipet from Download_Y script as I modified a little. Naming of the CSV file didnt work for me in the original script, so I changed it.
Zorro should print something like this:
Download_Y compiling........... FXCM.dll => FXCM plugin mounted Login FXCM.. at UTC 06-08 12:10 Downloading TLT.. Parse Historyhistory.csv.. 2017-06-07,125.239998,125.519997,124.790001,124.919998,124.919998,11951200 42893.00000000 125.2400 125.5200 124.7900 124.9200 11951200.0000 . 2017-06-06,125.690002,125.870003,125.400002,125.519997,125.519997,12841500 42892.00000000 125.6900 125.8700 125.4000 125.5200 12841500.0000 nTLT 3741 records read from YAHOO
Thank you very much for your reply. I tried your method in my MVO script with a Canadian stocks, I got this,
In my script, I set the StartDate=20160601, no EndDate, but the bars generated here is from 20160328 to 20170511. Also, the price of AC.TO is bit above 17$ but here it's 155.
Any thoughts?
Best
Jeff
Last edited by Jeff1228; 06/09/1703:17.
Re: Yahoo has changed their protocol
[Re: Jeff1228]
#466420 06/13/1712:2906/13/1712:29
So far we didnt need to have IB account to download from yahoo. I didnt change my fxcm login sttings and was able to download from yahoo.Did this change? Can somebody confirm? Can somebody confirm that beta version was also fixed not just normal version (i cant install normal version as my company laptop thinks it is a virus...)
Last edited by konorti; 06/21/1707:57.
Re: Yahoo has changed their protocol
[Re: konorti]
#466585 06/22/1708:2806/22/1708:28
Thanks! Yesterday I also wrote a code which easily downloads fromm Google, only Assets names should be enterred into the code. Enough to "test" the code. For me 1 minute was enough to download 80 assets' history:
I have a question : how to download data from alpha vantage I've tried this code but it says : { "Error Message": "the parameter apikey is invalid or missing. Please claim your free API key on (https://www.alphavantage.co/support/#api-key).
I have a question : how to download data from alpha vantage I've tried this code but it says : { "Error Message": "the parameter apikey is invalid or missing. Please claim your free API key on (https://www.alphavantage.co/support/#api-key).
function run() { NumYears=5;
assetHistory("MSFT",FROM_AV); }
THANKS
I think you need to apply for the key from AV and set it up in the .ini in Zorro folder.
Re: Yahoo has changed their protocol
[Re: firecrest]
#466606 06/23/1710:1506/23/1710:15
I have an another question, for example, I've tried to use the SP500 data from Yahoo to trade on SPX500 in FXCM. It's worked in backtest but when I try to use the script in live : it says : no asset data for ^GSPC
set(PRELOAD|LOGFILE|PLOTNOW); assetHistory("^GSPC",FROM_AV); asset("^GSPC"); PlotHeight1 = 800; // height of the chart with results PlotWidth = 2000;
///////////////////////////////// vars Price = series(priceClose()); vars high=series(priceHigh()); vars low=series(priceLow()); vars zerolag=series(KAMA(Price,10)); vars fastema = series(EMA(Price,50)); vars slowema = series(EMA(Price,80));
there is a solution to solve the problem, to use R but this solution takes more time, more memory so it's no efficient solution. May be you have a suggestion. Thanks
iF I use Yahoo (SP500 DAILY DATA) in my backtest, I obtain these résultats :
strategy compiling.......... Test: strategy ^GSPC 2008..2018 Monte Carlo Analysis... Median AR 9% Profit 4651$ MI 67$ DD 672$ Capital 841$ Trades 746 Win 76.8% Avg +757.5p Bars 2 AR 9% PF 2.58 SR 1.07 UI 3% R2 0.98 Chart... ok Copied to Clipboard!
If I use fxcm data (sp500 daily data ) , I obtain these results
strategy compiling...........
Test: strategy SPX500 2008..2018 Monte Carlo Analysis... Median AR 42% Profit 1850$ MI 25$ DD 694$ Capital 592$ Trades 620 Win 70.8% Avg +34.3p Bars 3 AR 50% PF 1.51 SR 0.51 UI 10% R2 0.87
It's clear that there a difference. In fact, I use open, high low and close data from Yahoo to do my backtest. These data are different from fxcm. That's why, I want use daily data for live trading in daily mode
Did you see any clear difference in trades in file: Logtesttrades.csv? I would test it also visually during only last months (to see trades on bars) what is difference between them. And maybe checking the history with file ZHistoryEditor.exe and trades from log file...
Re: Yahoo has changed their protocol
[Re: konorti]
#466734 06/29/1720:5106/29/1720:51
I haven't check the history but i think that the difference of performance is possibly caused by the nature of data, specially the high , low and close, and orders are not executed at the same moment,
I've changed like your recommandation : it returns Error 053: ^GSPC unavailable at this time Load NAS100 prices.. 5100 h0.00 0.00 0.00
Trade: model2 2017-06-30 Lookback period (145 bars) Error 030: asset ^GSPC missing in INITRUN Error 011: Function KAMA called with invalid parameters (no price) Error 011: Function EMA called with invalid parameters (no price)
Sorry I tried, it is the same problem for me. There is a way I think to import csv as data, and use as data and not as price history. I havent done that so I cant help.
Re: Yahoo has changed their protocol
[Re: konorti]
#466827 07/04/1708:3607/04/1708:36
I have been using AV for a while, I found the latest data (EOD) I could download from AV has been the data 2 trading days ago, not like other sites just a few hours after the trading hour is over I can get the latest EOD data. Has anyne found the same? What's wrong here?
Thank you
Re: Yahoo has changed their protocol
[Re: konorti]
#466894 07/06/1709:1607/06/1709:16
It has been mentioned in the past that data downloading through zorro from AV is slow because of the JSON format. According to their update website however the API now also allows to download csv files instead of JSON.
Re: Yahoo has changed their protocol
[Re: hatten]
#469376 11/13/1716:2211/13/1716:22