Maybe it would be useful for someone: this Python script I was using to get all Oanda instruments available for me (including RollLong and RollShort) for creating "All Oanda Assets List". Perhaps I did something not needed and already available, but I didn't find it. Please advise if it could be done smoother.
As it's described in https://www.zorro-trader.com/manual/en/account.htm in order to properly backtest the strategy on a certain account you need to use settings of this account. Every asset, which is available for your account, have certain characteristics. And as Oanda provides API, you could get these characteristics by running this script in Python.
However, the problem with Roll costs still remains as it is fixed for the backtest. So if your strategy holds a lot overnight it could lead to not realistic performance (too bad or too good). Maybe someone could advise how to mitigate this problem?
I took the liberty of extending your script to make it create a complete Assets.csv file, grouped by asset type, for all Oanda instruments, including the correct RollLong and RollShort values converted to the account currency. PIPCost is also calculated in account currency.
One thing to still be aware of is that Oanda appears to sometimes triple or quadruple the rollover rates even when it is not a Wednesday or Friday. I guess that could be taken care of by a script which runs a few times per week and compares the rates per instrument (manually or automatically).
Code
from collections import defaultdict
import pandas as pd
import oandapyV20
import oandapyV20.endpoints.accounts as accounts
import oandapyV20.endpoints.pricing as pricing
ACCOUNT_ID = 'XXX-XXX-XXXXXXX-XXX'
TOKEN = 'xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx-xxxxxxxxxxxxxxxxxxxxx'
ACCOUNT_CURRENCY = 'EUR'
OUTPUT_FILENAME = 'AssetsOanda.csv'
client = oandapyV20.API(access_token=TOKEN)
# Retrieve all instruments for account
r = accounts.AccountInstruments(accountID=ACCOUNT_ID)
rv = client.request(r)
data = pd.json_normalize(pd.DataFrame(rv)['instruments'])
# Retrieve pricing data for all instruments
instrument_list = list(data['name'])
r = pricing.PricingInfo(accountID=ACCOUNT_ID, params={'instruments': ','.join(map(str, instrument_list))})
rv = client.request(r)
prices = pd.json_normalize(pd.DataFrame(rv)['prices']).set_index('instrument').rename(columns={'instrument': 'Name'})
prices[['closeoutAsk', 'closeoutBid']] = prices[['closeoutAsk', 'closeoutBid']].astype(float)
prices['Price'] = prices[['closeoutAsk', 'closeoutBid']].mean(axis=1)
prices['Spread'] = prices['closeoutAsk'] - prices['closeoutBid']
# Rename columns, set instrument column as index, join with prices, and convert data types as necessary
data = data.rename(columns={'name': 'Name',
'minimumTradeSize': 'LotAmount'}). \
set_index('Name'). \
join(prices[['Price', 'Spread']])
data.index = data.index.str.replace('_','/') # Replace '_' with '/' in line with Zorro convention
convert_columns = ['LotAmount', 'marginRate', 'financing.longRate', 'financing.shortRate']
data[convert_columns] = data[convert_columns].astype(float)
data['currency'] = data.index.map(lambda c: c[-3:])
# Add Index as type, mainly for cosmetic reasons (where name ends in a number or in 'Index')
data.loc[(data['displayName'].str[-1].str.isnumeric()) | (data['displayName'].str[-5:]=='Index'),'type'] = 'INDEX'
# Store currency rates in dict, add inverse and necessary cross currency rates to be able to convert all
# assets to account currency (also add XAG as currency for XAU/XAG)
conversion_rate = data[data['type']=='CURRENCY']['Price'].to_dict() # All OANDA currency rates
conversion_rate.update({pair[-3:]+"/"+pair[:3]: 1/rate
for pair, rate in conversion_rate.items()}) # Inverse of existing currencies
conversion_rate[f"{ACCOUNT_CURRENCY}/{ACCOUNT_CURRENCY}"] = 1
assert 'USD/'+ACCOUNT_CURRENCY in conversion_rate, "Cannot calculate account currency rates"
conversion_rate['XAG/USD'] = data.loc['XAG/USD']['Price']
# Add missing currencies as cross currency via USD
conversion_rate.update({currency+'/'+ACCOUNT_CURRENCY: conversion_rate[currency+'/USD'] * conversion_rate['USD/'+ACCOUNT_CURRENCY]
for currency in
(currency for currency in data['currency']
if currency+'/'+ACCOUNT_CURRENCY not in conversion_rate)})
# Add additional columns, calculate PipCost based on conversion rates
data['PIP'] = 10.**data['pipLocation']
data['PIPCost'] = data['PIP'] * (data['currency']+'/'+ACCOUNT_CURRENCY).map(conversion_rate)
data['Leverage'] = (1/data['marginRate']).astype(int)
data['MarginCost'], data['Commission'] = 0, 0
data['Symbol'] = data.index
# Adjust for weekend rate inflation on Wednesdays/Fridays if necessary
# data[['financing.longRate', 'financing.shortRate']] = data[['financing.longRate', 'financing.shortRate']] / 4
# Calculate RollLong and RollShort in account currency, for the quantities used by Zorro
financing_pos_size = defaultdict(lambda: 1, {'CURRENCY': 10000}) # Everything but CURRENCY defaults to 1
data['RollLong'] = data['financing.longRate']/365 * \
data['Price'] * \
data['type'].map(financing_pos_size) * \
(data['currency']+'/'+ACCOUNT_CURRENCY).map(conversion_rate)
data['RollShort'] = data['financing.shortRate']/365 * \
data['Price'] * \
data['type'].map(financing_pos_size) * \
(data['currency']+'/'+ACCOUNT_CURRENCY).map(conversion_rate)
EXPORT_COLUMNS = ['Price', 'Spread', 'RollLong', 'RollShort',
'PIP', 'PIPCost', 'MarginCost', 'Leverage',
'LotAmount', 'Commission', 'Symbol']
DISPLAY_COLUMNS = ['type', 'displayName', 'Price', 'Spread',
'financing.longRate', 'financing.shortRate',
'RollLong', 'RollShort',
'PIP', 'PIPCost', 'MarginCost', 'Leverage',
'LotAmount', 'Commission']
data.sort_values(['type', 'Name'], inplace=True)
# Write CSV with group headers
data[EXPORT_COLUMNS].head(0).to_csv(OUTPUT_FILENAME)
max((open(OUTPUT_FILENAME, 'a').write(f"### {data_type}\n"),
data.loc[data_group, EXPORT_COLUMNS].to_csv(OUTPUT_FILENAME, float_format='%g', header=False, mode='a'))
for data_type, data_group in data.groupby('type').groups.items())
print(data[DISPLAY_COLUMNS].to_string())
Hi all. Thanks to kalmar and Morris! Below is a slightly updated version with the following additions:
- Note about Account Currency to be sure the user changes it if needed - Add timestamp to the output filename - Note of the change needed it it's a live account - Error handling if XAG is not valid (as for US residents)
I don't claim to be a Python programmer, so code review welcome from those who are!
Regards.
Code
from collections import defaultdict
from datetime import datetime
import pandas as pd
import oandapyV20
import oandapyV20.endpoints.accounts as accounts
import oandapyV20.endpoints.pricing as pricing
ACCOUNT_ID = 'XXX-XXX-XXXXXXX-XXX'
TOKEN = 'xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx-xxxxxxxxxxxxxxxxxxxxx'
ACCOUNT_CURRENCY = 'EUR' # or USD or whatever
OUTPUT_FILENAME = 'AssetsOanda-' + datetime.now().strftime("%Y-%m-%d-%H-%M-%S") + '.csv'
client = oandapyV20.API(access_token=TOKEN)
# For a live account:
# client = oandapyV20.API(access_token=TOKEN,environment='live')
# Retrieve all instruments for account
r = accounts.AccountInstruments(accountID=ACCOUNT_ID)
rv = client.request(r)
data = pd.json_normalize(pd.DataFrame(rv)['instruments'])
# Retrieve pricing data for all instruments
instrument_list = list(data['name'])
r = pricing.PricingInfo(accountID=ACCOUNT_ID, params={'instruments': ','.join(map(str, instrument_list))})
rv = client.request(r)
prices = pd.json_normalize(pd.DataFrame(rv)['prices']).set_index('instrument').rename(columns={'instrument': 'Name'})
prices[['closeoutAsk', 'closeoutBid']] = prices[['closeoutAsk', 'closeoutBid']].astype(float)
prices['Price'] = prices[['closeoutAsk', 'closeoutBid']].mean(axis=1)
prices['Spread'] = prices['closeoutAsk'] - prices['closeoutBid']
# Rename columns, set instrument column as index, join with prices, and convert data types as necessary
data = data.rename(columns={'name': 'Name',
'minimumTradeSize': 'LotAmount'}). \
set_index('Name'). \
join(prices[['Price', 'Spread']])
data.index = data.index.str.replace('_','/') # Replace '_' with '/' in line with Zorro convention
convert_columns = ['LotAmount', 'marginRate', 'financing.longRate', 'financing.shortRate']
data[convert_columns] = data[convert_columns].astype(float)
data['currency'] = data.index.map(lambda c: c[-3:])
# Add Index as type, mainly for cosmetic reasons (where name ends in a number or in 'Index')
data.loc[(data['displayName'].str[-1].str.isnumeric()) | (data['displayName'].str[-5:]=='Index'),'type'] = 'INDEX'
# Store currency rates in dict, add inverse and necessary cross currency rates to be able to convert all
# assets to account currency (also add XAG as currency for XAU/XAG)
conversion_rate = data[data['type']=='CURRENCY']['Price'].to_dict() # All OANDA currency rates
conversion_rate.update({pair[-3:]+"/"+pair[:3]: 1/rate
for pair, rate in conversion_rate.items()}) # Inverse of existing currencies
conversion_rate[f"{ACCOUNT_CURRENCY}/{ACCOUNT_CURRENCY}"] = 1
assert 'USD/'+ACCOUNT_CURRENCY in conversion_rate, "Cannot calculate account currency rates"
# Except, US types won't have XAG...
try:
conversion_rate['XAG/USD'] = data.loc['XAG/USD']['Price']
except KeyError:
print("No XAG/USD - is this a US account?!")
# Add missing currencies as cross currency via USD
conversion_rate.update({currency+'/'+ACCOUNT_CURRENCY: conversion_rate[currency+'/USD'] * conversion_rate['USD/'+ACCOUNT_CURRENCY]
for currency in
(currency for currency in data['currency']
if currency+'/'+ACCOUNT_CURRENCY not in conversion_rate)})
# Add additional columns, calculate PipCost based on conversion rates
data['PIP'] = 10.**data['pipLocation']
data['PIPCost'] = data['PIP'] * (data['currency']+'/'+ACCOUNT_CURRENCY).map(conversion_rate)
data['Leverage'] = (1/data['marginRate']).astype(int)
data['MarginCost'], data['Commission'] = 0, 0
data['Symbol'] = data.index
# Adjust for weekend rate inflation on Wednesdays/Fridays if necessary
# data[['financing.longRate', 'financing.shortRate']] = data[['financing.longRate', 'financing.shortRate']] / 4
# Calculate RollLong and RollShort in account currency, for the quantities used by Zorro
financing_pos_size = defaultdict(lambda: 1, {'CURRENCY': 10000}) # Everything but CURRENCY defaults to 1
data['RollLong'] = data['financing.longRate']/365 * \
data['Price'] * \
data['type'].map(financing_pos_size) * \
(data['currency']+'/'+ACCOUNT_CURRENCY).map(conversion_rate)
data['RollShort'] = data['financing.shortRate']/365 * \
data['Price'] * \
data['type'].map(financing_pos_size) * \
(data['currency']+'/'+ACCOUNT_CURRENCY).map(conversion_rate)
EXPORT_COLUMNS = ['Price', 'Spread', 'RollLong', 'RollShort',
'PIP', 'PIPCost', 'MarginCost', 'Leverage',
'LotAmount', 'Commission', 'Symbol']
DISPLAY_COLUMNS = ['type', 'displayName', 'Price', 'Spread',
'financing.longRate', 'financing.shortRate',
'RollLong', 'RollShort',
'PIP', 'PIPCost', 'MarginCost', 'Leverage',
'LotAmount', 'Commission']
data.sort_values(['type', 'Name'], inplace=True)
# Write CSV with group headers
data[EXPORT_COLUMNS].head(0).to_csv(OUTPUT_FILENAME)
max((open(OUTPUT_FILENAME, 'a').write(f"### {data_type}\n"),
data.loc[data_group, EXPORT_COLUMNS].to_csv(OUTPUT_FILENAME, float_format='%g', header=False, mode='a'))
for data_type, data_group in data.groupby('type').groups.items())
print(data[DISPLAY_COLUMNS].to_string())
Very useful -- thanks! (By the way, in order to check for the existence of XAG/USD, instead of try/except, you could use: if 'XAG/USD' in data.index ... -- but I realize this is not a Python forum...)
And kalmar, thanks for the pointer to finnhub.io. I look forward to checking it out!
@Eric: The code above is a Python script, so it won't run in Zorro. But it makes it easier to create the Zorro assets file for Oanda. To run it, just download and install Python and a few dependent packages.
I guess one could also use try/except for the initial account access to print a nice error message rather than just having an error dump if it's a live account... Wouldn't want the code to just fix itself and forge ahead on a live account, though - bad precedent!
I'm not sure this will run in Jupyter or in the shell (depending on how it is set up).
> I did use the pip option and installed both the oandapyV20 module and pandas
Good, that's an important step!
> SyntaxError: multiple statements found while compiling a single statement
That is probably because the shell usually just takes one statement at a time. It is not meant to execute scripts.
=> I would suggest to just run the script locally: python <scriptname>. Or, if your python is installed accordingly, just run the <script.py>, and it will be interpreted by python.