2 registered members (Imhotep, opm),
785
guests, and 4
spiders. |
Key:
Admin,
Global Mod,
Mod
|
|
|
Oanda instruments
#479654
04/15/20 21:49
04/15/20 21:49
|
Joined: Oct 2017
Posts: 56 Munich
kalmar
OP
Junior Member
|
OP
Junior Member
Joined: Oct 2017
Posts: 56
Munich
|
Hi all, Maybe it would be useful for someone: this Python script I was using to get all Oanda instruments available for me (including RollLong and RollShort) for creating "All Oanda Assets List". Perhaps I did something not needed and already available, but I didn't find it. Please advise if it could be done smoother. import json
import oandapyV20
import oandapyV20.endpoints.accounts as accounts
import pandas as pd
accountID = "XXX-XXX-XXXXXXX-XXX"
token = "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx-xxxxxxxxxxxxxxxxxxxxx"
client = oandapyV20.API(access_token=token)
r = accounts.AccountInstruments(accountID=accountID)
rv = client.request(r)
data = json.dumps(rv, indent=2)
frame = pd.json_normalize(pd.read_json(data)['instruments'])
needed_columns = ['name', 'type', 'displayName', 'minimumTradeSize','marginRate','financing.longRate','financing.shortRate']
frame[needed_columns].to_csv("Oanda_instruments.csv", index=False)
|
|
|
Re: Oanda instruments
[Re: kalmar]
#479911
05/04/20 15:55
05/04/20 15:55
|
Joined: Oct 2017
Posts: 56 Munich
kalmar
OP
Junior Member
|
OP
Junior Member
Joined: Oct 2017
Posts: 56
Munich
|
Hi Eric, I assume you have an Oanda account and Token. As it's described in https://www.zorro-trader.com/manual/en/account.htm in order to properly backtest the strategy on a certain account you need to use settings of this account. Every asset, which is available for your account, have certain characteristics. And as Oanda provides API, you could get these characteristics by running this script in Python. For oanda this will help as well: https://www.zorro-trader.com/manual/en/oanda.htmHowever, the problem with Roll costs still remains as it is fixed for the backtest. So if your strategy holds a lot overnight it could lead to not realistic performance (too bad or too good). Maybe someone could advise how to mitigate this problem? Thx, Kalmar
|
|
|
Re: Oanda instruments
[Re: kalmar]
#479995
05/11/20 19:54
05/11/20 19:54
|
Joined: May 2020
Posts: 27 Germany
Morris
Newbie
|
Newbie
Joined: May 2020
Posts: 27
Germany
|
Hello kalmar, Very useful indeed, thank you! I took the liberty of extending your script to make it create a complete Assets.csv file, grouped by asset type, for all Oanda instruments, including the correct RollLong and RollShort values converted to the account currency. PIPCost is also calculated in account currency. One thing to still be aware of is that Oanda appears to sometimes triple or quadruple the rollover rates even when it is not a Wednesday or Friday. I guess that could be taken care of by a script which runs a few times per week and compares the rates per instrument (manually or automatically). from collections import defaultdict
import pandas as pd
import oandapyV20
import oandapyV20.endpoints.accounts as accounts
import oandapyV20.endpoints.pricing as pricing
ACCOUNT_ID = 'XXX-XXX-XXXXXXX-XXX'
TOKEN = 'xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx-xxxxxxxxxxxxxxxxxxxxx'
ACCOUNT_CURRENCY = 'EUR'
OUTPUT_FILENAME = 'AssetsOanda.csv'
client = oandapyV20.API(access_token=TOKEN)
# Retrieve all instruments for account
r = accounts.AccountInstruments(accountID=ACCOUNT_ID)
rv = client.request(r)
data = pd.json_normalize(pd.DataFrame(rv)['instruments'])
# Retrieve pricing data for all instruments
instrument_list = list(data['name'])
r = pricing.PricingInfo(accountID=ACCOUNT_ID, params={'instruments': ','.join(map(str, instrument_list))})
rv = client.request(r)
prices = pd.json_normalize(pd.DataFrame(rv)['prices']).set_index('instrument').rename(columns={'instrument': 'Name'})
prices[['closeoutAsk', 'closeoutBid']] = prices[['closeoutAsk', 'closeoutBid']].astype(float)
prices['Price'] = prices[['closeoutAsk', 'closeoutBid']].mean(axis=1)
prices['Spread'] = prices['closeoutAsk'] - prices['closeoutBid']
# Rename columns, set instrument column as index, join with prices, and convert data types as necessary
data = data.rename(columns={'name': 'Name',
'minimumTradeSize': 'LotAmount'}). \
set_index('Name'). \
join(prices[['Price', 'Spread']])
data.index = data.index.str.replace('_','/') # Replace '_' with '/' in line with Zorro convention
convert_columns = ['LotAmount', 'marginRate', 'financing.longRate', 'financing.shortRate']
data[convert_columns] = data[convert_columns].astype(float)
data['currency'] = data.index.map(lambda c: c[-3:])
# Add Index as type, mainly for cosmetic reasons (where name ends in a number or in 'Index')
data.loc[(data['displayName'].str[-1].str.isnumeric()) | (data['displayName'].str[-5:]=='Index'),'type'] = 'INDEX'
# Store currency rates in dict, add inverse and necessary cross currency rates to be able to convert all
# assets to account currency (also add XAG as currency for XAU/XAG)
conversion_rate = data[data['type']=='CURRENCY']['Price'].to_dict() # All OANDA currency rates
conversion_rate.update({pair[-3:]+"/"+pair[:3]: 1/rate
for pair, rate in conversion_rate.items()}) # Inverse of existing currencies
conversion_rate[f"{ACCOUNT_CURRENCY}/{ACCOUNT_CURRENCY}"] = 1
assert 'USD/'+ACCOUNT_CURRENCY in conversion_rate, "Cannot calculate account currency rates"
conversion_rate['XAG/USD'] = data.loc['XAG/USD']['Price']
# Add missing currencies as cross currency via USD
conversion_rate.update({currency+'/'+ACCOUNT_CURRENCY: conversion_rate[currency+'/USD'] * conversion_rate['USD/'+ACCOUNT_CURRENCY]
for currency in
(currency for currency in data['currency']
if currency+'/'+ACCOUNT_CURRENCY not in conversion_rate)})
# Add additional columns, calculate PipCost based on conversion rates
data['PIP'] = 10.**data['pipLocation']
data['PIPCost'] = data['PIP'] * (data['currency']+'/'+ACCOUNT_CURRENCY).map(conversion_rate)
data['Leverage'] = (1/data['marginRate']).astype(int)
data['MarginCost'], data['Commission'] = 0, 0
data['Symbol'] = data.index
# Adjust for weekend rate inflation on Wednesdays/Fridays if necessary
# data[['financing.longRate', 'financing.shortRate']] = data[['financing.longRate', 'financing.shortRate']] / 4
# Calculate RollLong and RollShort in account currency, for the quantities used by Zorro
financing_pos_size = defaultdict(lambda: 1, {'CURRENCY': 10000}) # Everything but CURRENCY defaults to 1
data['RollLong'] = data['financing.longRate']/365 * \
data['Price'] * \
data['type'].map(financing_pos_size) * \
(data['currency']+'/'+ACCOUNT_CURRENCY).map(conversion_rate)
data['RollShort'] = data['financing.shortRate']/365 * \
data['Price'] * \
data['type'].map(financing_pos_size) * \
(data['currency']+'/'+ACCOUNT_CURRENCY).map(conversion_rate)
EXPORT_COLUMNS = ['Price', 'Spread', 'RollLong', 'RollShort',
'PIP', 'PIPCost', 'MarginCost', 'Leverage',
'LotAmount', 'Commission', 'Symbol']
DISPLAY_COLUMNS = ['type', 'displayName', 'Price', 'Spread',
'financing.longRate', 'financing.shortRate',
'RollLong', 'RollShort',
'PIP', 'PIPCost', 'MarginCost', 'Leverage',
'LotAmount', 'Commission']
data.sort_values(['type', 'Name'], inplace=True)
# Write CSV with group headers
data[EXPORT_COLUMNS].head(0).to_csv(OUTPUT_FILENAME)
max((open(OUTPUT_FILENAME, 'a').write(f"### {data_type}\n"),
data.loc[data_group, EXPORT_COLUMNS].to_csv(OUTPUT_FILENAME, float_format='%g', header=False, mode='a'))
for data_type, data_group in data.groupby('type').groups.items())
print(data[DISPLAY_COLUMNS].to_string())
|
|
|
Re: Oanda instruments
[Re: Morris]
#480260
05/28/20 22:16
05/28/20 22:16
|
Joined: Oct 2017
Posts: 56 Munich
kalmar
OP
Junior Member
|
OP
Junior Member
Joined: Oct 2017
Posts: 56
Munich
|
Hey Morris, Great stuff! I went through the code. Everything is clear. Thank you BTW I discovered for myself finnhub.io. They have a lot interesting stuff. Would be cool to have Zorro connection to it e.g. this snip gives you all the instruments from different forex providers they support (incl. Oanda) import fhub
hub = fhub.Session(token)
All_FX = pd.concat((hub.symbols(exch,kind='forex').assign(source = exch)
for exch in hub.exchanges('forex').forex), ignore_index = True)
|
|
|
Re: Oanda instruments
[Re: kalmar]
#480372
06/03/20 06:00
06/03/20 06:00
|
Joined: Jun 2013
Posts: 1,609
DdlV
Serious User
|
Serious User
Joined: Jun 2013
Posts: 1,609
|
Hi all. Thanks to kalmar and Morris! Below is a slightly updated version with the following additions: - Note about Account Currency to be sure the user changes it if needed - Add timestamp to the output filename - Note of the change needed it it's a live account - Error handling if XAG is not valid (as for US residents) I don't claim to be a Python programmer, so code review welcome from those who are! Regards.
from collections import defaultdict
from datetime import datetime
import pandas as pd
import oandapyV20
import oandapyV20.endpoints.accounts as accounts
import oandapyV20.endpoints.pricing as pricing
ACCOUNT_ID = 'XXX-XXX-XXXXXXX-XXX'
TOKEN = 'xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx-xxxxxxxxxxxxxxxxxxxxx'
ACCOUNT_CURRENCY = 'EUR' # or USD or whatever
OUTPUT_FILENAME = 'AssetsOanda-' + datetime.now().strftime("%Y-%m-%d-%H-%M-%S") + '.csv'
client = oandapyV20.API(access_token=TOKEN)
# For a live account:
# client = oandapyV20.API(access_token=TOKEN,environment='live')
# Retrieve all instruments for account
r = accounts.AccountInstruments(accountID=ACCOUNT_ID)
rv = client.request(r)
data = pd.json_normalize(pd.DataFrame(rv)['instruments'])
# Retrieve pricing data for all instruments
instrument_list = list(data['name'])
r = pricing.PricingInfo(accountID=ACCOUNT_ID, params={'instruments': ','.join(map(str, instrument_list))})
rv = client.request(r)
prices = pd.json_normalize(pd.DataFrame(rv)['prices']).set_index('instrument').rename(columns={'instrument': 'Name'})
prices[['closeoutAsk', 'closeoutBid']] = prices[['closeoutAsk', 'closeoutBid']].astype(float)
prices['Price'] = prices[['closeoutAsk', 'closeoutBid']].mean(axis=1)
prices['Spread'] = prices['closeoutAsk'] - prices['closeoutBid']
# Rename columns, set instrument column as index, join with prices, and convert data types as necessary
data = data.rename(columns={'name': 'Name',
'minimumTradeSize': 'LotAmount'}). \
set_index('Name'). \
join(prices[['Price', 'Spread']])
data.index = data.index.str.replace('_','/') # Replace '_' with '/' in line with Zorro convention
convert_columns = ['LotAmount', 'marginRate', 'financing.longRate', 'financing.shortRate']
data[convert_columns] = data[convert_columns].astype(float)
data['currency'] = data.index.map(lambda c: c[-3:])
# Add Index as type, mainly for cosmetic reasons (where name ends in a number or in 'Index')
data.loc[(data['displayName'].str[-1].str.isnumeric()) | (data['displayName'].str[-5:]=='Index'),'type'] = 'INDEX'
# Store currency rates in dict, add inverse and necessary cross currency rates to be able to convert all
# assets to account currency (also add XAG as currency for XAU/XAG)
conversion_rate = data[data['type']=='CURRENCY']['Price'].to_dict() # All OANDA currency rates
conversion_rate.update({pair[-3:]+"/"+pair[:3]: 1/rate
for pair, rate in conversion_rate.items()}) # Inverse of existing currencies
conversion_rate[f"{ACCOUNT_CURRENCY}/{ACCOUNT_CURRENCY}"] = 1
assert 'USD/'+ACCOUNT_CURRENCY in conversion_rate, "Cannot calculate account currency rates"
# Except, US types won't have XAG...
try:
conversion_rate['XAG/USD'] = data.loc['XAG/USD']['Price']
except KeyError:
print("No XAG/USD - is this a US account?!")
# Add missing currencies as cross currency via USD
conversion_rate.update({currency+'/'+ACCOUNT_CURRENCY: conversion_rate[currency+'/USD'] * conversion_rate['USD/'+ACCOUNT_CURRENCY]
for currency in
(currency for currency in data['currency']
if currency+'/'+ACCOUNT_CURRENCY not in conversion_rate)})
# Add additional columns, calculate PipCost based on conversion rates
data['PIP'] = 10.**data['pipLocation']
data['PIPCost'] = data['PIP'] * (data['currency']+'/'+ACCOUNT_CURRENCY).map(conversion_rate)
data['Leverage'] = (1/data['marginRate']).astype(int)
data['MarginCost'], data['Commission'] = 0, 0
data['Symbol'] = data.index
# Adjust for weekend rate inflation on Wednesdays/Fridays if necessary
# data[['financing.longRate', 'financing.shortRate']] = data[['financing.longRate', 'financing.shortRate']] / 4
# Calculate RollLong and RollShort in account currency, for the quantities used by Zorro
financing_pos_size = defaultdict(lambda: 1, {'CURRENCY': 10000}) # Everything but CURRENCY defaults to 1
data['RollLong'] = data['financing.longRate']/365 * \
data['Price'] * \
data['type'].map(financing_pos_size) * \
(data['currency']+'/'+ACCOUNT_CURRENCY).map(conversion_rate)
data['RollShort'] = data['financing.shortRate']/365 * \
data['Price'] * \
data['type'].map(financing_pos_size) * \
(data['currency']+'/'+ACCOUNT_CURRENCY).map(conversion_rate)
EXPORT_COLUMNS = ['Price', 'Spread', 'RollLong', 'RollShort',
'PIP', 'PIPCost', 'MarginCost', 'Leverage',
'LotAmount', 'Commission', 'Symbol']
DISPLAY_COLUMNS = ['type', 'displayName', 'Price', 'Spread',
'financing.longRate', 'financing.shortRate',
'RollLong', 'RollShort',
'PIP', 'PIPCost', 'MarginCost', 'Leverage',
'LotAmount', 'Commission']
data.sort_values(['type', 'Name'], inplace=True)
# Write CSV with group headers
data[EXPORT_COLUMNS].head(0).to_csv(OUTPUT_FILENAME)
max((open(OUTPUT_FILENAME, 'a').write(f"### {data_type}\n"),
data.loc[data_group, EXPORT_COLUMNS].to_csv(OUTPUT_FILENAME, float_format='%g', header=False, mode='a'))
for data_type, data_group in data.groupby('type').groups.items())
print(data[DISPLAY_COLUMNS].to_string())
|
|
|
|