Gamestudio Links
Zorro Links
Newest Posts
Blobsculptor tools and objects download here
by NeoDumont. 03/28/24 03:01
Issue with Multi-Core WFO Training
by aliswee. 03/24/24 20:20
Why Zorro supports up to 72 cores?
by Edgar_Herrera. 03/23/24 21:41
Zorro Trader GPT
by TipmyPip. 03/06/24 09:27
VSCode instead of SED
by 3run. 03/01/24 19:06
AUM Magazine
Latest Screens
The Bible Game
A psychological thriller game
SHADOW (2014)
DEAD TASTE
Who's Online Now
3 registered members (Edgar_Herrera, VoroneTZ, Akow), 973 guests, and 4 spiders.
Key: Admin, Global Mod, Mod
Newest Members
sakolin, rajesh7827, juergen_wue, NITRO_FOREVER, jack0roses
19043 Registered Users
Previous Thread
Next Thread
Print Thread
Rate Thread
Page 1 of 8 1 2 3 4 5 6 7 8
Yahoo has changed their protocol #465333
04/18/17 14:17
04/18/17 14:17
Joined: Jul 2000
Posts: 27,977
Frankfurt
jcl Offline OP

Chief Engineer
jcl  Offline OP

Chief Engineer

Joined: Jul 2000
Posts: 27,977
Frankfurt
We got several complaints today that downloading data from Yahoo does not work anymore. The reason is that Yahoo switched their servers to a new HTTPS protocol last weekend.

We've just uploaded a new beta version, 1.56.4, which downloads Yahoo data with the new protocol. You can get it from http://opserver.de/down/ZorrroBeta.zip.

Alternatively, you can still use your old version, but download Yahoo data from Quandl, f.i. assetHistory("YAHOO/AAPL",FROM_QUANDL);
You must register at Quandl for getting a Quandl key. Also, Quandl does support Stocks, but not all ETFs in their Yahoo database.

Update: For downloading all the Z8 assets with the beta version, you can use a small script like this:

Code:
void main()
{
	assetList("AssetsZ8");
	while(asset(loop(Assets)))
		assetHistory(Asset,FROM_YAHOO);
}


Re: Yahoo has changed their protocol [Re: jcl] #465664
05/08/17 21:40
05/08/17 21:40
Joined: Mar 2015
Posts: 25
J
jmb Offline
Newbie
jmb  Offline
Newbie
J

Joined: Mar 2015
Posts: 25
It seems that some other things have changed at IB.

The Zorro manual states (or it did last time I looked) that you need to be subscribed to "US Value Bundle PLUS" data subscription for Z8.

To make this work, I found I needed these:
- AMEX (Network B/CTA)
- NASDAQ (Network C/UTP)

I cancelled my entry for "US Value Bundle PLUS".

Re: Yahoo has changed their protocol [Re: jmb] #465896
05/17/17 01:02
05/17/17 01:02
Joined: Jan 2017
Posts: 64
J
Jeff1228 Offline
Junior Member
Jeff1228  Offline
Junior Member
J

Joined: Jan 2017
Posts: 64
Still have some problems downloading from Yahoo.

I tested the Heatmap script built-in Zorro. I tried both V1.56 and V1.58, both of them carshed like below:

Heatmap compiling..............
Bad date format - can't parse Historyhistory.csv
0 AEIS bars read
0 AMAT bars read
More assets..
0 AMD bars read
Time 00:01:06
Error 111: Crash in script
Time 00:01:06

What did I do wrong?

By the way, does the data from Yahoo have volume information? What's the right function to get the volume info.

Thank you

Jeff

Re: Yahoo has changed their protocol [Re: Jeff1228] #465901
05/17/17 08:15
05/17/17 08:15
Joined: Jul 2000
Posts: 27,977
Frankfurt
jcl Offline OP

Chief Engineer
jcl  Offline OP

Chief Engineer

Joined: Jul 2000
Posts: 27,977
Frankfurt
Yes, Yahoo data has volume info. Or had, at least - look into the "history.csv" with a HTML browser and you'll get the answer for the new download problems. frown

Re: Yahoo has changed their protocol [Re: jcl] #465905
05/17/17 09:07
05/17/17 09:07
Joined: Jan 2017
Posts: 64
J
Jeff1228 Offline
Junior Member
Jeff1228  Offline
Junior Member
J

Joined: Jan 2017
Posts: 64
IE and Chrome can't open it.

Which is the function to get volume info? Like I use priceClose(0) for the close price.

Thank you

Jeff

Re: Yahoo has changed their protocol [Re: Jeff1228] #465907
05/17/17 09:47
05/17/17 09:47
Joined: Jan 2017
Posts: 64
J
Jeff1228 Offline
Junior Member
Jeff1228  Offline
Junior Member
J

Joined: Jan 2017
Posts: 64
"history.csv" comes with Zorro, I didn't modify it, It should have been working.

Jeff

Last edited by Jeff1228; 05/17/17 10:48.
Re: Yahoo has changed their protocol [Re: Jeff1228] #465913
05/17/17 11:25
05/17/17 11:25
Joined: Jul 2000
Posts: 27,977
Frankfurt
jcl Offline OP

Chief Engineer
jcl  Offline OP

Chief Engineer

Joined: Jul 2000
Posts: 27,977
Frankfurt
Had you managed to open it with your browser, you had seen this message:

Yahoo
Will be right back...
Thank you for your patience.
Our engineers are working quickly to resolve the issue.


Re: Yahoo has changed their protocol [Re: jcl] #465924
05/18/17 01:05
05/18/17 01:05
Joined: Jan 2017
Posts: 64
J
Jeff1228 Offline
Junior Member
Jeff1228  Offline
Junior Member
J

Joined: Jan 2017
Posts: 64
No, I tried both IE and Chrome, didn't work out. I asked one of my friend tried, not either. Which browser do you use?

Jeff

Re: Yahoo has changed their protocol [Re: Jeff1228] #465930
05/18/17 06:47
05/18/17 06:47
Joined: Jul 2015
Posts: 8
J
Jens Offline
Newbie
Jens  Offline
Newbie
J

Joined: Jul 2015
Posts: 8
Jeff, you have to rename the "history.csv" to "history.html" in order to open it in Chrome or IE.

Re: Yahoo has changed their protocol [Re: Jens] #465934
05/18/17 08:06
05/18/17 08:06
Joined: Jul 2000
Posts: 27,977
Frankfurt
jcl Offline OP

Chief Engineer
jcl  Offline OP

Chief Engineer

Joined: Jul 2000
Posts: 27,977
Frankfurt
I have been informed meanwhile that it's unlikely that the "Yahoo engineers" will fix it in the next time.

We'll have to look for an alternative to Yahoo for ETF prices via API. Unfortunately, all I found so far is not free.

Re: Yahoo has changed their protocol [Re: jcl] #465935
05/18/17 08:15
05/18/17 08:15
Joined: Jul 2015
Posts: 8
J
Jens Offline
Newbie
Jens  Offline
Newbie
J

Joined: Jul 2015
Posts: 8

Re: Yahoo has changed their protocol [Re: Jens] #465936
05/18/17 08:40
05/18/17 08:40
Joined: Jul 2000
Posts: 27,977
Frankfurt
jcl Offline OP

Chief Engineer
jcl  Offline OP

Chief Engineer

Joined: Jul 2000
Posts: 27,977
Frankfurt
That's their browser button. It works only in a browser and cannot be used for automated download.

Re: Yahoo has changed their protocol [Re: jcl] #465945
05/18/17 11:06
05/18/17 11:06
Joined: Jul 2000
Posts: 27,977
Frankfurt
jcl Offline OP

Chief Engineer
jcl  Offline OP

Chief Engineer

Joined: Jul 2000
Posts: 27,977
Frankfurt
Ok, we found that you can download the same ETF data from Google. But I don't know how long, their API is also depreciated. For the moment, it works. So when you want to test or run Z8, run this script before:

Code:
void main()
{
	string Code;
	int N = assetList("AssetsZ8");
	while(--N >= 0) 
	{
		string URL = strf("https://www.google.com/finance/historical?q=%s&startdate=01-Jan-2000&output=csv",Assets[N]);
		string Content = http_transfer(URL,0);
		file_write("History\\history.csv",Content,0);
		free(Content);
		dataNew(1,0,7);
		dataParse(1,"%d-%b-%y,f3,f1,f2,f4,f6","History\\history.csv");
		printf(" %s",Assets[N]);
		dataSave(1,strf("%s.t6",Assets[N]));
  }
}




Re: Yahoo has changed their protocol [Re: jcl] #465962
05/18/17 17:51
05/18/17 17:51
Joined: Dec 2013
Posts: 568
Fuerth, DE
Sphin Offline
User
Sphin  Offline
User

Joined: Dec 2013
Posts: 568
Fuerth, DE
Quote:
it's unlikely that the "Yahoo engineers" will fix it in the next time.

Very unlikely:
https://forums.yahoo.net/t5/Yahoo-Finance-help/Is-Yahoo-Finance-API-broken/td-p/250503/page/3:
"This feature was discontinued by the Finance team and they will not be reintroducing that functionality."

This has a massive impact, imagine only how many R analysis' work with Yahoo API ...

Re: Yahoo has changed their protocol [Re: Sphin] #465970
05/19/17 04:04
05/19/17 04:04
Joined: Jan 2017
Posts: 64
J
Jeff1228 Offline
Junior Member
Jeff1228  Offline
Junior Member
J

Joined: Jan 2017
Posts: 64
I have account and data subscription with IB, and I run your scripts Heatmap and MVO, can I simple change "assetHistory(Names[N],FROM_YAHOO)" to "assetHistory(Names[N],FROM_IB)"? Would it work?

Is your code a seperate script I should run before I run other scripts? If so, should I run it every time before I run others? or I should put it somewhere in your Heatmap and MVO script?

Can I get stocks data from Google?

Last edited by Jeff1228; 05/19/17 04:17.
Re: Yahoo has changed their protocol [Re: Jeff1228] #465971
05/19/17 04:28
05/19/17 04:28
Joined: Jan 2017
Posts: 64
J
Jeff1228 Offline
Junior Member
Jeff1228  Offline
Junior Member
J

Joined: Jan 2017
Posts: 64
I tried, and found this


google compiling...........

Parse Historyhistory.csv... UCTT
Parse Historyhistory.csv... UVE
Parse Historyhistory.csv... BSX
Parse Historyhistory.csv... LOW. FRME
Parse Historyhistory.csv... EQIX
Parse Historyhistory.csv... MTN
Parse Historyhistory.csv... UGI
Parse Historyhistory.csv... TREX
Error 111: Crash in script: main()


My assetList comprises all stocks.

Re: Yahoo has changed their protocol [Re: jcl] #465972
05/19/17 05:40
05/19/17 05:40
Joined: Jul 2016
Posts: 51
H
hatten Offline
Junior Member
hatten  Offline
Junior Member
H

Joined: Jul 2016
Posts: 51
This script for google is unfortunately not working. It yields the following:

googleDownload compiling......

Parse Historyhistroy.csv... XRT
Parse Historyhistory.csv... XLY
Bad - can't parse Historyhistory.csv XLV
Parse Historyhistory.csv... XLU XLP XLK XLI XLF
Parse Historyhistory.csv... XBI
Parse Historyhistory.csv... UPRO
Error 111: Crash in script: main()

Interesting enough, a second run does not yield the same
log:

googleDownload compiling...........

Parse Historyhistory.csv... XRT
Error 111: Crash in script: main()

Without the download functionality, the usefullness of zorro is very limited if any. What options do we have?

Re: Yahoo has changed their protocol [Re: hatten] #465973
05/19/17 06:38
05/19/17 06:38
Joined: Jul 2000
Posts: 27,977
Frankfurt
jcl Offline OP

Chief Engineer
jcl  Offline OP

Chief Engineer

Joined: Jul 2000
Posts: 27,977
Frankfurt
Dowloading from Google works for me with the default Z8 assets, but maybe not with old Zorro versions or when you have added other assets. If Google does not support the asset, the parse function returns 0 and the script crashes. Printing an error message instead is left as an exercise to the reader.

Google is no final solution since they will also discontinue the API some day. I'm afraid the final solution will possibly not be free.

-The Google download can be run before any script with Yahoo download. The script will detect that the data is already up to date. Google has also stocks, but major stocks are no problem anyway since you get them free from Quandl. The problem are ETFs.

Re: Yahoo has changed their protocol [Re: jcl] #465976
05/19/17 07:27
05/19/17 07:27
Joined: Jan 2017
Posts: 64
J
Jeff1228 Offline
Junior Member
Jeff1228  Offline
Junior Member
J

Joined: Jan 2017
Posts: 64
Thank you for your always prompt reply. Like I said, I have IB data subscription, how can I use it?

Thanks

Re: Yahoo has changed their protocol [Re: Jeff1228] #465977
05/19/17 07:36
05/19/17 07:36
Joined: Jul 2000
Posts: 27,977
Frankfurt
jcl Offline OP

Chief Engineer
jcl  Offline OP

Chief Engineer

Joined: Jul 2000
Posts: 27,977
Frankfurt
Not directly. IB delivers only 1 year history.

Re: Yahoo has changed their protocol [Re: jcl] #465978
05/19/17 07:55
05/19/17 07:55
Joined: Jan 2017
Posts: 64
J
Jeff1228 Offline
Junior Member
Jeff1228  Offline
Junior Member
J

Joined: Jan 2017
Posts: 64
So the Quandl is the only possible way to do this by subscribing it?

Re: Yahoo has changed their protocol [Re: jcl] #465979
05/19/17 08:26
05/19/17 08:26
Joined: Jul 2000
Posts: 27,977
Frankfurt
jcl Offline OP

Chief Engineer
jcl  Offline OP

Chief Engineer

Joined: Jul 2000
Posts: 27,977
Frankfurt
Not for stocks, but for ETFs. But we'll try to find an alternative solution.

Here's a modified Google download script that does not crash anymore when an asset is not found:
Code:
void main()
{
	int N = assetList("AssetsZ8");
	while(--N >= 0) 
	{
		string URL = strf("https://www.google.com/finance/historical?q=%s&startdate=01-Jan-2000&output=csv",Assets[N]);
		string Content = http_transfer(URL,0);
		if(!Content) continue;
		file_write("History\\history.csv",Content,0);
		dataNew(1,0,7);
		if(!dataParse(1,"%d-%b-%y,f3,f1,f2,f4,f6","History\\history.csv"))
			continue;
		printf(" %s",Assets[N]);
		dataSave(1,strf("History\\%s.t6",Assets[N]));
	}
}


Re: Yahoo has changed their protocol [Re: jcl] #465997
05/19/17 10:50
05/19/17 10:50
Joined: Jul 2016
Posts: 51
H
hatten Offline
Junior Member
hatten  Offline
Junior Member
H

Joined: Jul 2016
Posts: 51
While the google download script now indeed no longer causes a crash, there is a large discrepancy in the test results of the Z8 system with the google data. The drawdown is more substantial (The previous test was made with the 1.58.0 beta version, whose result I fortunately still had in the log directory), AR was about 18% with leverage 2 and now it is down to 13%. Is googles data adjusted for dividends, splits etc.?

Re: Yahoo has changed their protocol [Re: hatten] #465998
05/19/17 12:30
05/19/17 12:30
Joined: Jul 2000
Posts: 27,977
Frankfurt
jcl Offline OP

Chief Engineer
jcl  Offline OP

Chief Engineer

Joined: Jul 2000
Posts: 27,977
Frankfurt
Only for splits. AFAIK dividends are not included. This affects the result substantially. Also the Google data has a lot more outliers and gaps than the Yahoo data. So this is only a temporary solution.

We will implement a final solution in the next Zorro update.

Re: Yahoo has changed their protocol [Re: jcl] #466008
05/19/17 19:39
05/19/17 19:39
Joined: Jul 2016
Posts: 51
H
hatten Offline
Junior Member
hatten  Offline
Junior Member
H

Joined: Jul 2016
Posts: 51
If I read their webpage correctly eoddata delivers historical end-of-the-day data for a very affordable one time payment.

Moreover it says:
"If you purchase access to 5 years historical end of day quote data then you can download all 5 years and in 3 years time there will be 8 years for you to download. Once you have purchased access, you can download your historical end of day data as many times as you like. If your PC ever crashes and you loose your data simply download it again."

So as time progress you get also access to new data.

Re: Yahoo has changed their protocol [Re: hatten] #466011
05/20/17 00:58
05/20/17 00:58
Joined: Apr 2014
Posts: 482
Sydney, Australia
B
boatman Offline
Senior Member
boatman  Offline
Senior Member
B

Joined: Apr 2014
Posts: 482
Sydney, Australia
For anyone using the quantmod library in R, Joshua Ulrich (the package's author) released a workaround that you can get by updating quantmod to the development version. In R, do either:

remotes::install_github("joshuaulrich/quantmod", ref="157_yahoo_502")
or
devtools::install_github("joshuaulrich/quantmod", ref="157_yahoo_502")

More info on the GitHub page:
https://github.com/joshuaulrich/quantmod/issues/157

This works for me. I was able to successfully use quantmod::getSymbols() to query the Yahoo API.

Re: Yahoo has changed their protocol [Re: boatman] #466015
05/20/17 19:52
05/20/17 19:52
Joined: Dec 2013
Posts: 568
Fuerth, DE
Sphin Offline
User
Sphin  Offline
User

Joined: Dec 2013
Posts: 568
Fuerth, DE
I search for a Yahoo API substitute also for another reason than Zorro and I stumbled upon a site, I don't know really what to think of it. alphavantage.co offers free (you can apply a free API key) intraday, daily, weekly and monthly prices for US stock, ETFs etc. Their history e.g. of TLT goes back to 2002, of SPY back to 2001. The output format is JSON but this is to handle I guess. I did not verify the quality of their prices so far.
They declare their aim as "democratizing access to institution-grade financial analytics" but they have no address or company name on their site and even the domain name alphavantage.co is registered by an agent it seems. The IP address 216.239.36.21 of their server belongs to a Google net block. As said, I don't know how reputable they are.

Re: Yahoo has changed their protocol [Re: Sphin] #466049
05/23/17 03:29
05/23/17 03:29
Joined: Jan 2017
Posts: 64
J
Jeff1228 Offline
Junior Member
Jeff1228  Offline
Junior Member
J

Joined: Jan 2017
Posts: 64
Still no solution?

Re: Yahoo has changed their protocol [Re: Jeff1228] #466097
05/24/17 18:13
05/24/17 18:13
Joined: Dec 2013
Posts: 568
Fuerth, DE
Sphin Offline
User
Sphin  Offline
User

Joined: Dec 2013
Posts: 568
Fuerth, DE
Also an elegant solution by Ed0906. Following are code snippets of his approach. It works for now, the question is how long ...

Re: Yahoo has changed their protocol [Re: Sphin] #466107
05/25/17 08:18
05/25/17 08:18
Joined: Jan 2017
Posts: 64
J
Jeff1228 Offline
Junior Member
Jeff1228  Offline
Junior Member
J

Joined: Jan 2017
Posts: 64
Thank you for the info, seems there are some solutions out there. Unfortunately I am not a programmer, hope Zorro will catch up.

Jeff

Re: Yahoo has changed their protocol [Re: Jeff1228] #466113
05/25/17 15:19
05/25/17 15:19
Joined: Jul 2015
Posts: 8
J
Jens Offline
Newbie
Jens  Offline
Newbie
J

Joined: Jul 2015
Posts: 8
This is a really nice solution and working. I tested the C# class.

Re: Yahoo has changed their protocol [Re: Jens] #466118
05/25/17 18:49
05/25/17 18:49
Joined: Dec 2013
Posts: 568
Fuerth, DE
Sphin Offline
User
Sphin  Offline
User

Joined: Dec 2013
Posts: 568
Fuerth, DE
I use the php solution in my website and it's also working for now. But no one knows if and when Y will tighten its security's features. The cookie's validity of 1 year is e.g. very convenient ...

Re: Yahoo has changed their protocol [Re: Sphin] #466127
05/26/17 06:40
05/26/17 06:40
Joined: Jul 2015
Posts: 8
J
Jens Offline
Newbie
Jens  Offline
Newbie
J

Joined: Jul 2015
Posts: 8
Another way is to use the .NET webbrowser control in order to load the whole website from yahoo, boerse-frankfurt etc. Then we get a HTML document object and can walk through the tables and it's rows and cells to get the quotes. This is working for me, too.

Re: Yahoo has changed their protocol [Re: Jens] #466155
05/27/17 17:08
05/27/17 17:08
Joined: Jul 2015
Posts: 8
J
Jens Offline
Newbie
Jens  Offline
Newbie
J

Joined: Jul 2015
Posts: 8
For many ETFs I get historical prices only for 2 years via the described Yahoo interface. For the same ETFs I get 5 years or more at boerse-frankfurt.de.

Re: Yahoo has changed their protocol [Re: jcl] #466157
05/27/17 23:39
05/27/17 23:39
Joined: Nov 2016
Posts: 103
NSW
V
vinsom Offline
Member
vinsom  Offline
Member
V

Joined: Nov 2016
Posts: 103
NSW
I made this .Net exe from previous links to C# code
source code attached.

Put the exe in zorro folder.
Then call it with this in Zorro


var dataFromYahoo(string Code, int mode)
{
string Format;
if (mode == FROM_YAHOO | UNADJUSTED)
Format = "%Y-%m-%d,f3,f1,f2,,f4,f6";//UnadjAdj Close
else if (mode == FROM_YAHOO)
Format = "%Y-%m-%d,f3,f1,f2,f4,,f6";//Adj Close

exec("YahooFinanceAPI.exe", Code, 1);
//exec("YahooFinanceAPI.exe", "AAPL 1",1);
dataNew(1, 0, 7);
int numRecord = dataParse(1, Format, "History\history.csv");
printf("n%s %d records read from YAHOO", Code, numRecord);
dataSave(1, strf("%s.t6", Code));
return numRecord;
}

Attached Files
YahooFinanceAPI_CS.zip (236 downloads)
Re: Yahoo has changed their protocol [Re: vinsom] #466214
06/01/17 09:09
06/01/17 09:09
Joined: Jan 2017
Posts: 64
J
Jeff1228 Offline
Junior Member
Jeff1228  Offline
Junior Member
J

Joined: Jan 2017
Posts: 64
Originally Posted By: vinsom
I made this .Net exe from previous links to C# code
source code attached.

Put the exe in zorro folder.
Then call it with this in Zorro


var dataFromYahoo(string Code, int mode)
{
string Format;
if (mode == FROM_YAHOO | UNADJUSTED)
Format = "%Y-%m-%d,f3,f1,f2,,f4,f6";//UnadjAdj Close
else if (mode == FROM_YAHOO)
Format = "%Y-%m-%d,f3,f1,f2,f4,,f6";//Adj Close

exec("YahooFinanceAPI.exe", Code, 1);
//exec("YahooFinanceAPI.exe", "AAPL 1",1);
dataNew(1, 0, 7);
int numRecord = dataParse(1, Format, "Historyhistory.csv");
printf("n%s %d records read from YAHOO", Code, numRecord);
dataSave(1, strf("%s.t6", Code));
return numRecord;
}



Thank you but I don't quite understand this.

Where is the exe code?

What's the purpose of the code here? Should I put it into the script hwere I use FROM_YAHOO?

What's the purpose of the attachment?

I am new here but I really need this working.

Jeff

Re: Yahoo has changed their protocol [Re: Jeff1228] #466225
06/01/17 21:06
06/01/17 21:06
Joined: Nov 2016
Posts: 103
NSW
V
vinsom Offline
Member
vinsom  Offline
Member
V

Joined: Nov 2016
Posts: 103
NSW
You need to compile the attached code with MS Visual studio (can't attach exe).
This will generate the exe YahooFinanceAPI.exe.
Put this in the Zorro folder.
Then get the history with the zorro code snippet above, that will call the exe
generated, with exec("YahooFinanceAPI.exe", Code, 1);.
Cheers

Re: Yahoo has changed their protocol [Re: vinsom] #466237
06/02/17 08:05
06/02/17 08:05
Joined: Jul 2016
Posts: 27
K
konorti Offline
Newbie
konorti  Offline
Newbie
K

Joined: Jul 2016
Posts: 27
Sorry I dont have MS Visual Studio. Can you please upload the exe to wetransfer.com or similar site? Thanks!
Update: i tried to build with msbuild but didnt compile because of error.

Last edited by konorti; 06/02/17 19:30.
Re: Yahoo has changed their protocol [Re: konorti] #466247
06/02/17 23:54
06/02/17 23:54
Joined: Nov 2016
Posts: 103
NSW
V
vinsom Offline
Member
vinsom  Offline
Member
V

Joined: Nov 2016
Posts: 103
NSW
Here it is.
https://we.tl/Jfs1kuwNRJ

need the .Net framework to work.
Download it from microsoft.

Regards

Re: Yahoo has changed their protocol [Re: vinsom] #466249
06/03/17 06:05
06/03/17 06:05
Joined: Jul 2016
Posts: 27
K
konorti Offline
Newbie
konorti  Offline
Newbie
K

Joined: Jul 2016
Posts: 27
Thanks! I also modified the "Download" strategy to use your exe.

Attached Files
Download_Y.c (333 downloads)
Re: Yahoo has changed their protocol [Re: konorti] #466302
06/07/17 06:08
06/07/17 06:08
Joined: Jan 2017
Posts: 64
J
Jeff1228 Offline
Junior Member
Jeff1228  Offline
Junior Member
J

Joined: Jan 2017
Posts: 64
Hi all,

Can I use the google script provided by jcl to download Canadian stocks history from Yahoo, I tried but got this

google compiling..............

Bad - can't parse Historyhistory.csv
Time 00:01:14

Last time I used an assetList with all US stocks, it worked.

I copied the link "https://www.google.com/finance/historical?q=AC.TO&startdate=01-Jan-2000" into the browser, actually it worked and I got the data up to date.

What's wrong here?

Thank you

Jeff

Last edited by Jeff1228; 06/07/17 11:10.
Re: Yahoo has changed their protocol [Re: konorti] #466305
06/07/17 08:37
06/07/17 08:37
Joined: Jan 2017
Posts: 64
J
Jeff1228 Offline
Junior Member
Jeff1228  Offline
Junior Member
J

Joined: Jan 2017
Posts: 64
Originally Posted By: konorti
Thanks! I also modified the "Download" strategy to use your exe.


Hi,

I am using MVO script and an assetlist with all Canadian stocks. I tried your method to download data from YAHOO. I put the .exe to the Zorro fold and your "var dataFromYahoo(string Code, int mode)" snippet as a function into the MVO script ahead of run function, changed orignal "assetHistory(Names[N],FROM_YAHOO);" to "dataFromYahoo(Names[N],FROM_YAHOO);". When I run it, for each of the assets, it seemed that a blank window pooped up waiting for some input, but I could not type in anything. When I crossed it out, Zorro seemed to download data. But at the end, the data is all old, not updated!

What's wrong with my script?

Thank you

Last edited by Jeff1228; 06/07/17 09:11.
Re: Yahoo has changed their protocol [Re: vinsom] #466319
06/08/17 08:41
06/08/17 08:41
Joined: Jan 2017
Posts: 64
J
Jeff1228 Offline
Junior Member
Jeff1228  Offline
Junior Member
J

Joined: Jan 2017
Posts: 64
Originally Posted By: vinsom
Here it is.
https://we.tl/Jfs1kuwNRJ

need the .Net framework to work.
Download it from microsoft.

Regards


Hi vinsom,

Are you Dennis Lee on Github? Is this .exe for Yahoo's cookie and token? I have tried but did not work out. I use win10 with .net built-in.

What went wrong?

Thanks

Re: Yahoo has changed their protocol [Re: Jeff1228] #466322
06/08/17 12:14
06/08/17 12:14
Joined: Jul 2016
Posts: 27
K
konorti Offline
Newbie
konorti  Offline
Newbie
K

Joined: Jul 2016
Posts: 27
I think pop-up empty window is fine. This works the same as original Download script that you have to write asset name and press "Load Yahoo Data" then. Try the the code snipet from Download_Y script as I modified a little. Naming of the CSV file didnt work for me in the original script, so I changed it.

Zorro should print something like this:

Download_Y compiling...........
FXCM.dll => FXCM plugin mounted
Login FXCM.. at UTC 06-08 12:10
Downloading TLT..
Parse Historyhistory.csv..
2017-06-07,125.239998,125.519997,124.790001,124.919998,124.919998,11951200
42893.00000000 125.2400 125.5200 124.7900 124.9200 11951200.0000 .
2017-06-06,125.690002,125.870003,125.400002,125.519997,125.519997,12841500
42892.00000000 125.6900 125.8700 125.4000 125.5200 12841500.0000 nTLT 3741 records read from YAHOO

Last edited by konorti; 06/08/17 12:20.
Re: Yahoo has changed their protocol [Re: konorti] #466329
06/09/17 02:25
06/09/17 02:25
Joined: Jan 2017
Posts: 64
J
Jeff1228 Offline
Junior Member
Jeff1228  Offline
Junior Member
J

Joined: Jan 2017
Posts: 64
Originally Posted By: konorti
I think pop-up empty window is fine. This works the same as original Download script that you have to write asset name and press "Load Yahoo Data" then. Try the the code snipet from Download_Y script as I modified a little. Naming of the CSV file didnt work for me in the original script, so I changed it.

Zorro should print something like this:

Download_Y compiling...........
FXCM.dll => FXCM plugin mounted
Login FXCM.. at UTC 06-08 12:10
Downloading TLT..
Parse Historyhistory.csv..
2017-06-07,125.239998,125.519997,124.790001,124.919998,124.919998,11951200
42893.00000000 125.2400 125.5200 124.7900 124.9200 11951200.0000 .
2017-06-06,125.690002,125.870003,125.400002,125.519997,125.519997,12841500
42892.00000000 125.6900 125.8700 125.4000 125.5200 12841500.0000 nTLT 3741 records read from YAHOO



Thank you very much for your reply. I tried your method in my MVO script with a Canadian stocks, I got this,


MVO compiling...........
Parse Historyhistory.csv..
8-Jun-17,155.25,155.54,154.40,154.99,21250798
39447.00000000 155.2500 155.5400 154.4000 21250798.0000 .
7-Jun-17,155.02,155.98,154.48,155.37,21069647
39082.00000000 155.0200 155.9800 154.4800 21069647.0000
AC.TO 4001 records read from YAHOO
284 Bars 2016-03-28..2017-05-11 generated
Test: MVO AC.TO 2016..2017
Assets CATEST666.csv


In my script, I set the StartDate=20160601, no EndDate, but the bars generated here is from 20160328 to 20170511. Also, the price of AC.TO is bit above 17$ but here it's 155.

Any thoughts?

Best

Jeff


Last edited by Jeff1228; 06/09/17 03:17.
Re: Yahoo has changed their protocol [Re: Jeff1228] #466420
06/13/17 12:29
06/13/17 12:29
Joined: Jan 2017
Posts: 64
J
Jeff1228 Offline
Junior Member
Jeff1228  Offline
Junior Member
J

Joined: Jan 2017
Posts: 64
Finally a solution is working, thank you team Zorro

Re: Yahoo has changed their protocol [Re: Jeff1228] #466421
06/13/17 13:12
06/13/17 13:12
Joined: Dec 2016
Posts: 71
F
firecrest Offline
Junior Member
firecrest  Offline
Junior Member
F

Joined: Dec 2016
Posts: 71
Hi Jeff1228,

Would you be able to share the solution? I am trading Z8. Thanks.

Re: Yahoo has changed their protocol [Re: firecrest] #466433
06/14/17 02:28
06/14/17 02:28
Joined: Jan 2017
Posts: 64
J
Jeff1228 Offline
Junior Member
Jeff1228  Offline
Junior Member
J

Joined: Jan 2017
Posts: 64
Zorro releases a new version 1.59, then you use Alphavantage, then you go!

Re: Yahoo has changed their protocol [Re: Jeff1228] #466541
06/20/17 08:30
06/20/17 08:30
Joined: Jul 2016
Posts: 27
K
konorti Offline
Newbie
konorti  Offline
Newbie
K

Joined: Jul 2016
Posts: 27
I downloaded beta version 1.59.4 but I cant download history with it.
It freezes in the message:

Downloading AGG..

Re: Yahoo has changed their protocol [Re: konorti] #466545
06/20/17 12:35
06/20/17 12:35
Joined: Jun 2016
Posts: 30
france
S
stephane97490 Offline
Newbie
stephane97490  Offline
Newbie
S

Joined: Jun 2016
Posts: 30
france
Hi,
Can Zorro team improve a script for downloading data from https://stooq.com/
it canbe useful instead using Yahoo or else ?

Re: Yahoo has changed their protocol [Re: konorti] #466546
06/20/17 12:53
06/20/17 12:53
Joined: Dec 2016
Posts: 71
F
firecrest Offline
Junior Member
firecrest  Offline
Junior Member
F

Joined: Dec 2016
Posts: 71
Originally Posted By: konorti
I downloaded beta version 1.59.4 but I cant download history with it.
It freezes in the message:

Downloading AGG..


Maybe you need to log onto IB before downloading.

Re: Yahoo has changed their protocol [Re: firecrest] #466560
06/21/17 07:30
06/21/17 07:30
Joined: Jul 2016
Posts: 27
K
konorti Offline
Newbie
konorti  Offline
Newbie
K

Joined: Jul 2016
Posts: 27
So far we didnt need to have IB account to download from yahoo. I didnt change my fxcm login sttings and was able to download from yahoo.Did this change? Can somebody confirm? Can somebody confirm that beta version was also fixed not just normal version (i cant install normal version as my company laptop thinks it is a virus...)

Last edited by konorti; 06/21/17 07:57.
Re: Yahoo has changed their protocol [Re: konorti] #466585
06/22/17 08:28
06/22/17 08:28
Joined: Nov 2016
Posts: 66
GreenBoat Offline
Junior Member
GreenBoat  Offline
Junior Member

Joined: Nov 2016
Posts: 66
Yahoo download doesn't work anymore, that's what this thread is all about.

Re: Yahoo has changed their protocol [Re: GreenBoat] #466591
06/22/17 10:24
06/22/17 10:24
Joined: Jul 2000
Posts: 27,977
Frankfurt
jcl Offline OP

Chief Engineer
jcl  Offline OP

Chief Engineer

Joined: Jul 2000
Posts: 27,977
Frankfurt
Beta 1.59.4 had a bug with AV download. Use 1.59.5.

Re: Yahoo has changed their protocol [Re: jcl] #466600
06/22/17 13:09
06/22/17 13:09
Joined: Jul 2016
Posts: 27
K
konorti Offline
Newbie
konorti  Offline
Newbie
K

Joined: Jul 2016
Posts: 27
Thanks!
Yesterday I also wrote a code which easily downloads fromm Google, only Assets names should be enterred into the code. Enough to "test" the code. For me 1 minute was enough to download 80 assets' history:
Quote:

int download (string ass)
{
string Code = ass;
string URL = strf("http://www.google.com/finance/historical?q=%s&startdate=01-Jan-2000&output=csv",Code);
string Content = http_transfer(URL,0);

file_write("Historyhistory.csv",Content,0);
dataNew(1,0,7);
if(dataParse(1,"%d-%b-%y,f3,f1,f2,f4,f6","Historyhistory.csv"))
dataSave(1,strf("History%s.t6",Code));
}

void main()
{

while(loop( // Here you can list all assets
"AGG",
"ALFA",
"AMZA",
// ....
))
download(Loop1);


}




Last edited by konorti; 06/22/17 13:11.
Re: Yahoo has changed their protocol [Re: jcl] #466603
06/22/17 20:16
06/22/17 20:16
Joined: Jun 2016
Posts: 30
france
S
stephane97490 Offline
Newbie
stephane97490  Offline
Newbie
S

Joined: Jun 2016
Posts: 30
france
I have a question : how to download data from alpha vantage
I've tried this code but it says : {
"Error Message": "the parameter apikey is invalid or missing. Please claim your free API key on (https://www.alphavantage.co/support/#api-key).

function run()
{
NumYears=5;

assetHistory("MSFT",FROM_AV);
}

THANKS

Re: Yahoo has changed their protocol [Re: stephane97490] #466605
06/23/17 03:48
06/23/17 03:48
Joined: Dec 2016
Posts: 71
F
firecrest Offline
Junior Member
firecrest  Offline
Junior Member
F

Joined: Dec 2016
Posts: 71
Originally Posted By: stephane97490
I have a question : how to download data from alpha vantage
I've tried this code but it says : {
"Error Message": "the parameter apikey is invalid or missing. Please claim your free API key on (https://www.alphavantage.co/support/#api-key).

function run()
{
NumYears=5;

assetHistory("MSFT",FROM_AV);
}

THANKS


I think you need to apply for the key from AV and set it up in the .ini in Zorro folder.

Re: Yahoo has changed their protocol [Re: firecrest] #466606
06/23/17 10:15
06/23/17 10:15
Joined: Jun 2016
Posts: 30
france
S
stephane97490 Offline
Newbie
stephane97490  Offline
Newbie
S

Joined: Jun 2016
Posts: 30
france
THANKS for your help, it's ok for AV

Re: Yahoo has changed their protocol [Re: firecrest] #466609
06/23/17 12:13
06/23/17 12:13
Joined: Jun 2016
Posts: 30
france
S
stephane97490 Offline
Newbie
stephane97490  Offline
Newbie
S

Joined: Jun 2016
Posts: 30
france
I have an another question, for example, I've tried to use the SP500 data from Yahoo to trade on SPX500 in FXCM. It's worked in backtest but when I try to use the script in live : it says :
no asset data for ^GSPC

the script is :

function run()
{
StartDate = 20100101;
EndDate = 20171231;
BarPeriod = 60*24; // 2 hour
LookBack = 600;

set(PRELOAD|LOGFILE|PLOTNOW);
assetHistory("^GSPC",FROM_AV);
asset("^GSPC");
PlotHeight1 = 800; // height of the chart with results
PlotWidth = 2000;

/////////////////////////////////
vars Price = series(priceClose());
vars high=series(priceHigh());
vars low=series(priceLow());
vars zerolag=series(KAMA(Price,10));
vars fastema = series(EMA(Price,50));
vars slowema = series(EMA(Price,80));

asset("SPX500");
if(fastema[0]>slowema[0])
{
enterLong();

}

if(fastema[0]<slowema[0])
{
exitLong();

}


}




there is a solution to solve the problem, to use R but this solution takes more time, more memory so it's no efficient solution.
May be you have a suggestion.
Thanks

Re: Yahoo has changed their protocol [Re: stephane97490] #466638
06/24/17 19:59
06/24/17 19:59
Joined: Jul 2016
Posts: 27
K
konorti Offline
Newbie
konorti  Offline
Newbie
K

Joined: Jul 2016
Posts: 27
Sorry maybe this is a dumb question. Why not download history from FXCM?

Last edited by konorti; 06/24/17 19:59.
Re: Yahoo has changed their protocol [Re: konorti] #466659
06/26/17 17:15
06/26/17 17:15
Joined: Jun 2016
Posts: 30
france
S
stephane97490 Offline
Newbie
stephane97490  Offline
Newbie
S

Joined: Jun 2016
Posts: 30
france
iF I use Yahoo (SP500 DAILY DATA) in my backtest, I obtain these résultats :


strategy compiling..........
Test: strategy ^GSPC 2008..2018
Monte Carlo Analysis... Median AR 9%
Profit 4651$ MI 67$ DD 672$ Capital 841$
Trades 746 Win 76.8% Avg +757.5p Bars 2
AR 9% PF 2.58 SR 1.07 UI 3% R2 0.98
Chart... ok
Copied to Clipboard!


If I use fxcm data (sp500 daily data ) , I obtain these results

strategy compiling...........

Test: strategy SPX500 2008..2018
Monte Carlo Analysis... Median AR 42%
Profit 1850$ MI 25$ DD 694$ Capital 592$
Trades 620 Win 70.8% Avg +34.3p Bars 3
AR 50% PF 1.51 SR 0.51 UI 10% R2 0.87



It's clear that there a difference. In fact, I use open, high low and close data from Yahoo to do my backtest. These data are different from fxcm. That's why, I want use daily data for live trading in daily mode

Re: Yahoo has changed their protocol [Re: stephane97490] #466733
06/29/17 20:38
06/29/17 20:38
Joined: Jul 2016
Posts: 27
K
konorti Offline
Newbie
konorti  Offline
Newbie
K

Joined: Jul 2016
Posts: 27
Did you see any clear difference in trades in file: Logtesttrades.csv?
I would test it also visually during only last months (to see trades on bars) what is difference between them.
And maybe checking the history with file ZHistoryEditor.exe and trades from log file...

Re: Yahoo has changed their protocol [Re: konorti] #466734
06/29/17 20:51
06/29/17 20:51
Joined: Jun 2016
Posts: 30
france
S
stephane97490 Offline
Newbie
stephane97490  Offline
Newbie
S

Joined: Jun 2016
Posts: 30
france
I haven't check the history but i think that the difference of performance is possibly caused by the nature of data, specially the high , low and close, and orders are not executed at the same moment,

Re: Yahoo has changed their protocol [Re: stephane97490] #466735
06/29/17 21:13
06/29/17 21:13
Joined: Jul 2016
Posts: 27
K
konorti Offline
Newbie
konorti  Offline
Newbie
K

Joined: Jul 2016
Posts: 27
Did you enter into AssetsFix.csv the line:
^GSPC,50,0.1,0,0,0.01,0.01,0,2,1,0.02,

Re: Yahoo has changed their protocol [Re: konorti] #466750
06/30/17 15:56
06/30/17 15:56
Joined: Jun 2016
Posts: 30
france
S
stephane97490 Offline
Newbie
stephane97490  Offline
Newbie
S

Joined: Jun 2016
Posts: 30
france
I've changed like your recommandation : it returns
Error 053: ^GSPC unavailable at this time
Load NAS100 prices.. 5100 h0.00 0.00 0.00

Trade: model2 2017-06-30
Lookback period (145 bars)
Error 030: asset ^GSPC missing in INITRUN
Error 011: Function KAMA called with invalid parameters (no price)
Error 011: Function EMA called with invalid parameters (no price)



the script is :

function run()
{
NumYears=15;
StartDate = 20100101;
EndDate = 20171231;
BarPeriod = 60*24;
LookBack = 145;
set(PRELOAD);


//asset("^GSPC");
//if(is(INITRUN))
assetHistory("^GSPC",FROM_AV);
asset("^GSPC");
vars Price = series(priceClose());
vars high=series(priceHigh());
vars low=series(priceLow());
vars zerolag=series(KAMA(Price,10));
vars fastema = series(EMA(Price,50));
vars slowema = series(EMA(Price,80));
/////////////////////////////////
vars slowema = series(EMA(Price,80));
asset("NAS100");
if(fastema[0]>slowema[0])
{
enterLong();
}
if(fastema[0]<slowema[0])
{
exitLong();
}

printf("%.2f ",zerolag[0]);
printf("%.2f ",fastema[0]);
printf("%.2f ",slowema[0]);
}

Re: Yahoo has changed their protocol [Re: stephane97490] #466757
06/30/17 18:05
06/30/17 18:05
Joined: Jul 2016
Posts: 27
K
konorti Offline
Newbie
konorti  Offline
Newbie
K

Joined: Jul 2016
Posts: 27
Next week i will try to replicate your code to google history as i dont have zorro paid version to access AV history to see what is the problem.

Re: Yahoo has changed their protocol [Re: konorti] #466826
07/04/17 07:16
07/04/17 07:16
Joined: Jul 2016
Posts: 27
K
konorti Offline
Newbie
konorti  Offline
Newbie
K

Joined: Jul 2016
Posts: 27
Sorry I tried, it is the same problem for me. There is a way I think to import csv as data, and use as data and not as price history. I havent done that so I cant help.

Re: Yahoo has changed their protocol [Re: konorti] #466827
07/04/17 08:36
07/04/17 08:36
Joined: Jan 2017
Posts: 64
J
Jeff1228 Offline
Junior Member
Jeff1228  Offline
Junior Member
J

Joined: Jan 2017
Posts: 64
I have been using AV for a while, I found the latest data (EOD) I could download from AV has been the data 2 trading days ago, not like other sites just a few hours after the trading hour is over I can get the latest EOD data. Has anyne found the same? What's wrong here?

Thank you

Re: Yahoo has changed their protocol [Re: konorti] #466894
07/06/17 09:16
07/06/17 09:16
Joined: Jun 2016
Posts: 30
france
S
stephane97490 Offline
Newbie
stephane97490  Offline
Newbie
S

Joined: Jun 2016
Posts: 30
france
thanks

Re: Yahoo has changed their protocol [Re: Jeff1228] #466895
07/06/17 09:17
07/06/17 09:17
Joined: Jun 2016
Posts: 30
france
S
stephane97490 Offline
Newbie
stephane97490  Offline
Newbie
S

Joined: Jun 2016
Posts: 30
france
I've noticed that AV depends on yahoo data, so when yahoo isn't working, AV doesn't deliver or updates their data

Re: Yahoo has changed their protocol [Re: stephane97490] #467626
08/17/17 20:59
08/17/17 20:59
Joined: Jul 2016
Posts: 51
H
hatten Offline
Junior Member
hatten  Offline
Junior Member
H

Joined: Jul 2016
Posts: 51
It has been mentioned in the past that data downloading through zorro from AV is slow because of the JSON format. According to their update website however the API now also allows to download csv files instead of JSON.

Re: Yahoo has changed their protocol [Re: hatten] #469376
11/13/17 16:22
11/13/17 16:22
Joined: Jul 2016
Posts: 93
Düsseldorf, Germany
M
mhdus Offline
Junior Member
mhdus  Offline
Junior Member
M

Joined: Jul 2016
Posts: 93
Düsseldorf, Germany
Barchart has launched an alternative:
https://www.barchart.com/solutions/press...nance-api-users
@jcl, could you consider this for integrating into the Zorro data source options?

Last edited by mhdus; 11/13/17 16:22.
Re: Yahoo has changed their protocol [Re: mhdus] #469377
11/13/17 16:31
11/13/17 16:31
Joined: Sep 2017
Posts: 235
H
Hredot Offline
Member
Hredot  Offline
Member
H

Joined: Sep 2017
Posts: 235
@mhdus
I might be wrong, but isn't 2 years of daily history a ridiculously small time frame which makes this service essentially useless?

Re: Yahoo has changed their protocol [Re: Hredot] #469378
11/13/17 16:43
11/13/17 16:43
Joined: Jul 2016
Posts: 93
Düsseldorf, Germany
M
mhdus Offline
Junior Member
mhdus  Offline
Junior Member
M

Joined: Jul 2016
Posts: 93
Düsseldorf, Germany
Ok, I overlooked this... frown Then maybe not really an alternative but rather an interesting additonal source as it has intraday data and futures data.

Re: Yahoo has changed their protocol [Re: Hredot] #469379
11/13/17 18:26
11/13/17 18:26
Joined: Aug 2017
Posts: 58
P
pascalx Offline
Junior Member
pascalx  Offline
Junior Member
P

Joined: Aug 2017
Posts: 58
"2 years of daily history"
"3 months of n-minute history"

Does not look very useful indeed.

Re: Yahoo has changed their protocol [Re: pascalx] #482661
03/14/21 21:39
03/14/21 21:39
Joined: Feb 2021
Posts: 1
L
LorraineJones Offline
Guest
LorraineJones  Offline
Guest
L

Joined: Feb 2021
Posts: 1
I've been using this one for the last few months and you can get the last 10 years of history.

Page 1 of 8 1 2 3 4 5 6 7 8

Moderated by  Petra 

Gamestudio download | chip programmers | Zorro platform | shop | Data Protection Policy

oP group Germany GmbH | Birkenstr. 25-27 | 63549 Ronneburg / Germany | info (at) opgroup.de

Powered by UBB.threads™ PHP Forum Software 7.7.1