Gamestudio Links
Zorro Links
Newest Posts
Zorro Trader GPT
by TipmyPip. 03/06/24 09:27
VSCode instead of SED
by 3run. 03/01/24 19:06
Deeplearning Script
by wolfi. 02/26/24 12:46
Sam Foster Sound | Experienced Game Composer for Hire
by titanicpiano14. 02/22/24 16:22
AssetAdd() vs. modern asset list?
by jcl. 02/21/24 15:01
How many still using A8
by Aku_Aku. 02/20/24 12:18
AUM Magazine
Latest Screens
The Bible Game
A psychological thriller game
SHADOW (2014)
DEAD TASTE
Who's Online Now
2 registered members (TipmyPip, 1 invisible), 595 guests, and 7 spiders.
Key: Admin, Global Mod, Mod
Newest Members
sakolin, rajesh7827, juergen_wue, NITRO_FOREVER, jack0roses
19043 Registered Users
Previous Thread
Next Thread
Print Thread
Rate Thread
Page 1 of 2 1 2
Oanda instruments #479654
04/15/20 21:49
04/15/20 21:49
Joined: Oct 2017
Posts: 56
Munich
K
kalmar Offline OP
Junior Member
kalmar  Offline OP
Junior Member
K

Joined: Oct 2017
Posts: 56
Munich
Hi all,

Maybe it would be useful for someone: this Python script I was using to get all Oanda instruments available for me (including RollLong and RollShort) for creating "All Oanda Assets List". Perhaps I did something not needed and already available, but I didn't find it. Please advise if it could be done smoother.

Code
import json
import oandapyV20
import oandapyV20.endpoints.accounts as accounts
import pandas as pd

accountID = "XXX-XXX-XXXXXXX-XXX"
token = "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx-xxxxxxxxxxxxxxxxxxxxx"
client = oandapyV20.API(access_token=token)

r = accounts.AccountInstruments(accountID=accountID)
rv = client.request(r)
data = json.dumps(rv, indent=2)
frame = pd.json_normalize(pd.read_json(data)['instruments'])
needed_columns = ['name', 'type', 'displayName', 'minimumTradeSize','marginRate','financing.longRate','financing.shortRate']
frame[needed_columns].to_csv("Oanda_instruments.csv", index=False)

Re: Oanda instruments [Re: kalmar] #479894
05/03/20 17:26
05/03/20 17:26
Joined: Apr 2020
Posts: 14
E
ESR Offline
Newbie
ESR  Offline
Newbie
E

Joined: Apr 2020
Posts: 14
kalmar, thanks for this.

Would you mind providing a step-by-step walkthrough for implementing this. because I am new to Zorro?

TIA,

~eric

Re: Oanda instruments [Re: kalmar] #479911
05/04/20 15:55
05/04/20 15:55
Joined: Oct 2017
Posts: 56
Munich
K
kalmar Offline OP
Junior Member
kalmar  Offline OP
Junior Member
K

Joined: Oct 2017
Posts: 56
Munich
Hi Eric,

I assume you have an Oanda account and Token.

As it's described in https://www.zorro-trader.com/manual/en/account.htm in order to properly backtest the strategy on a certain account you need to use settings of this account. Every asset, which is available for your account, have certain characteristics. And as Oanda provides API, you could get these characteristics by running this script in Python.

For oanda this will help as well: https://www.zorro-trader.com/manual/en/oanda.htm

However, the problem with Roll costs still remains as it is fixed for the backtest. So if your strategy holds a lot overnight it could lead to not realistic performance (too bad or too good). Maybe someone could advise how to mitigate this problem?

Thx,

Kalmar

Re: Oanda instruments [Re: kalmar] #479995
05/11/20 19:54
05/11/20 19:54
Joined: May 2020
Posts: 27
Germany
M
Morris Offline
Newbie
Morris  Offline
Newbie
M

Joined: May 2020
Posts: 27
Germany
Hello kalmar,

Very useful indeed, thank you!

I took the liberty of extending your script to make it create a complete Assets.csv file, grouped by asset type, for all Oanda instruments, including the correct RollLong and RollShort values converted to the account currency. PIPCost is also calculated in account currency.

One thing to still be aware of is that Oanda appears to sometimes triple or quadruple the rollover rates even when it is not a Wednesday or Friday. I guess that could be taken care of by a script which runs a few times per week and compares the rates per instrument (manually or automatically).

Code
from collections import defaultdict
import pandas as pd

import oandapyV20
import oandapyV20.endpoints.accounts as accounts
import oandapyV20.endpoints.pricing as pricing

ACCOUNT_ID = 'XXX-XXX-XXXXXXX-XXX'
TOKEN = 'xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx-xxxxxxxxxxxxxxxxxxxxx'
ACCOUNT_CURRENCY = 'EUR'
OUTPUT_FILENAME = 'AssetsOanda.csv'

client = oandapyV20.API(access_token=TOKEN)

# Retrieve all instruments for account
r = accounts.AccountInstruments(accountID=ACCOUNT_ID)
rv = client.request(r)
data = pd.json_normalize(pd.DataFrame(rv)['instruments'])

# Retrieve pricing data for all instruments
instrument_list = list(data['name'])
r = pricing.PricingInfo(accountID=ACCOUNT_ID, params={'instruments': ','.join(map(str, instrument_list))})
rv = client.request(r)
prices = pd.json_normalize(pd.DataFrame(rv)['prices']).set_index('instrument').rename(columns={'instrument': 'Name'})
prices[['closeoutAsk', 'closeoutBid']] = prices[['closeoutAsk', 'closeoutBid']].astype(float)
prices['Price'] = prices[['closeoutAsk', 'closeoutBid']].mean(axis=1)
prices['Spread'] = prices['closeoutAsk'] - prices['closeoutBid']

# Rename columns, set instrument column as index, join with prices, and convert data types as necessary
data = data.rename(columns={'name': 'Name',
                            'minimumTradeSize': 'LotAmount'}). \
            set_index('Name'). \
            join(prices[['Price', 'Spread']])
data.index = data.index.str.replace('_','/')               # Replace '_' with '/' in line with Zorro convention
convert_columns = ['LotAmount', 'marginRate', 'financing.longRate', 'financing.shortRate']
data[convert_columns] = data[convert_columns].astype(float)
data['currency'] = data.index.map(lambda c: c[-3:])

# Add Index as type, mainly for cosmetic reasons (where name ends in a number or in 'Index')
data.loc[(data['displayName'].str[-1].str.isnumeric()) | (data['displayName'].str[-5:]=='Index'),'type'] = 'INDEX'

# Store currency rates in dict, add inverse and necessary cross currency rates to be able to convert all
# assets to account currency (also add XAG as currency for XAU/XAG)
conversion_rate = data[data['type']=='CURRENCY']['Price'].to_dict()     # All OANDA currency rates
conversion_rate.update({pair[-3:]+"/"+pair[:3]: 1/rate
                        for pair, rate in conversion_rate.items()})     # Inverse of existing currencies
conversion_rate[f"{ACCOUNT_CURRENCY}/{ACCOUNT_CURRENCY}"] = 1
assert 'USD/'+ACCOUNT_CURRENCY in conversion_rate, "Cannot calculate account currency rates"
conversion_rate['XAG/USD'] = data.loc['XAG/USD']['Price']
# Add missing currencies as cross currency via USD
conversion_rate.update({currency+'/'+ACCOUNT_CURRENCY: conversion_rate[currency+'/USD'] * conversion_rate['USD/'+ACCOUNT_CURRENCY]
                        for currency in
                            (currency for currency in data['currency']
                             if currency+'/'+ACCOUNT_CURRENCY not in conversion_rate)})

# Add additional columns, calculate PipCost based on conversion rates
data['PIP'] = 10.**data['pipLocation']
data['PIPCost'] = data['PIP'] * (data['currency']+'/'+ACCOUNT_CURRENCY).map(conversion_rate)
data['Leverage'] = (1/data['marginRate']).astype(int)
data['MarginCost'], data['Commission'] = 0, 0
data['Symbol'] = data.index

# Adjust for weekend rate inflation on Wednesdays/Fridays if necessary
# data[['financing.longRate', 'financing.shortRate']] = data[['financing.longRate', 'financing.shortRate']] / 4

# Calculate RollLong and RollShort in account currency, for the quantities used by Zorro
financing_pos_size = defaultdict(lambda: 1, {'CURRENCY': 10000})    # Everything but CURRENCY defaults to 1
data['RollLong'] = data['financing.longRate']/365 * \
                   data['Price'] * \
                   data['type'].map(financing_pos_size) * \
                  (data['currency']+'/'+ACCOUNT_CURRENCY).map(conversion_rate)
data['RollShort'] = data['financing.shortRate']/365 * \
                    data['Price'] * \
                    data['type'].map(financing_pos_size) * \
                   (data['currency']+'/'+ACCOUNT_CURRENCY).map(conversion_rate)

EXPORT_COLUMNS = ['Price', 'Spread', 'RollLong', 'RollShort',
                  'PIP', 'PIPCost', 'MarginCost', 'Leverage',
                  'LotAmount', 'Commission', 'Symbol']
DISPLAY_COLUMNS = ['type', 'displayName', 'Price', 'Spread',
                   'financing.longRate', 'financing.shortRate',
                   'RollLong', 'RollShort',
                   'PIP', 'PIPCost', 'MarginCost', 'Leverage',
                   'LotAmount', 'Commission']
data.sort_values(['type', 'Name'], inplace=True)

# Write CSV with group headers
data[EXPORT_COLUMNS].head(0).to_csv(OUTPUT_FILENAME)
max((open(OUTPUT_FILENAME, 'a').write(f"### {data_type}\n"),
          data.loc[data_group, EXPORT_COLUMNS].to_csv(OUTPUT_FILENAME, float_format='%g', header=False, mode='a'))
    for data_type, data_group in data.groupby('type').groups.items())

print(data[DISPLAY_COLUMNS].to_string())

Re: Oanda instruments [Re: Morris] #480258
05/28/20 21:17
05/28/20 21:17
Joined: Apr 2020
Posts: 14
E
ESR Offline
Newbie
ESR  Offline
Newbie
E

Joined: Apr 2020
Posts: 14
@Morris thanks for the code to create an Oanda Assets file.

When I run this script I get:

_OandaAssets compiling.......
Error in 'line 2:
'from' undeclared identifier
< from collections import defaultdict
>.

What did I miss?

~eric

Re: Oanda instruments [Re: kalmar] #480259
05/28/20 21:29
05/28/20 21:29
Joined: Apr 2020
Posts: 14
E
ESR Offline
Newbie
ESR  Offline
Newbie
E

Joined: Apr 2020
Posts: 14
Thanks Kalmar,

What I meant to ask is how do you run the script.

~eric

Re: Oanda instruments [Re: Morris] #480260
05/28/20 22:16
05/28/20 22:16
Joined: Oct 2017
Posts: 56
Munich
K
kalmar Offline OP
Junior Member
kalmar  Offline OP
Junior Member
K

Joined: Oct 2017
Posts: 56
Munich
Hey Morris,

Great stuff! I went through the code. Everything is clear.

Thank you

BTW I discovered for myself finnhub.io. They have a lot interesting stuff. Would be cool to have Zorro connection to it laugh

e.g. this snip gives you all the instruments from different forex providers they support (incl. Oanda)

Code
import fhub
hub = fhub.Session(token)
All_FX = pd.concat((hub.symbols(exch,kind='forex').assign(source = exch) 
                        for exch in hub.exchanges('forex').forex), ignore_index = True) 

Re: Oanda instruments [Re: kalmar] #480372
06/03/20 06:00
06/03/20 06:00
Joined: Jun 2013
Posts: 1,609
D
DdlV Offline
Serious User
DdlV  Offline
Serious User
D

Joined: Jun 2013
Posts: 1,609
Hi all. Thanks to kalmar and Morris! Below is a slightly updated version with the following additions:

- Note about Account Currency to be sure the user changes it if needed
- Add timestamp to the output filename
- Note of the change needed it it's a live account
- Error handling if XAG is not valid (as for US residents)

I don't claim to be a Python programmer, so code review welcome from those who are!

Regards.

Code
from collections import defaultdict
from datetime import datetime
import pandas as pd

import oandapyV20
import oandapyV20.endpoints.accounts as accounts
import oandapyV20.endpoints.pricing as pricing

ACCOUNT_ID = 'XXX-XXX-XXXXXXX-XXX'
TOKEN = 'xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx-xxxxxxxxxxxxxxxxxxxxx'
ACCOUNT_CURRENCY = 'EUR'    # or USD or whatever
OUTPUT_FILENAME = 'AssetsOanda-' + datetime.now().strftime("%Y-%m-%d-%H-%M-%S") + '.csv'

client = oandapyV20.API(access_token=TOKEN)
# For a live account:
# client = oandapyV20.API(access_token=TOKEN,environment='live')

# Retrieve all instruments for account
r = accounts.AccountInstruments(accountID=ACCOUNT_ID)
rv = client.request(r)
data = pd.json_normalize(pd.DataFrame(rv)['instruments'])

# Retrieve pricing data for all instruments
instrument_list = list(data['name'])
r = pricing.PricingInfo(accountID=ACCOUNT_ID, params={'instruments': ','.join(map(str, instrument_list))})
rv = client.request(r)
prices = pd.json_normalize(pd.DataFrame(rv)['prices']).set_index('instrument').rename(columns={'instrument': 'Name'})
prices[['closeoutAsk', 'closeoutBid']] = prices[['closeoutAsk', 'closeoutBid']].astype(float)
prices['Price'] = prices[['closeoutAsk', 'closeoutBid']].mean(axis=1)
prices['Spread'] = prices['closeoutAsk'] - prices['closeoutBid']

# Rename columns, set instrument column as index, join with prices, and convert data types as necessary
data = data.rename(columns={'name': 'Name',
                            'minimumTradeSize': 'LotAmount'}). \
            set_index('Name'). \
            join(prices[['Price', 'Spread']])
data.index = data.index.str.replace('_','/')               # Replace '_' with '/' in line with Zorro convention
convert_columns = ['LotAmount', 'marginRate', 'financing.longRate', 'financing.shortRate']
data[convert_columns] = data[convert_columns].astype(float)
data['currency'] = data.index.map(lambda c: c[-3:])

# Add Index as type, mainly for cosmetic reasons (where name ends in a number or in 'Index')
data.loc[(data['displayName'].str[-1].str.isnumeric()) | (data['displayName'].str[-5:]=='Index'),'type'] = 'INDEX'

# Store currency rates in dict, add inverse and necessary cross currency rates to be able to convert all
# assets to account currency (also add XAG as currency for XAU/XAG)
conversion_rate = data[data['type']=='CURRENCY']['Price'].to_dict()     # All OANDA currency rates
conversion_rate.update({pair[-3:]+"/"+pair[:3]: 1/rate
                        for pair, rate in conversion_rate.items()})     # Inverse of existing currencies
conversion_rate[f"{ACCOUNT_CURRENCY}/{ACCOUNT_CURRENCY}"] = 1
assert 'USD/'+ACCOUNT_CURRENCY in conversion_rate, "Cannot calculate account currency rates"

# Except, US types won't have XAG...
try:
    conversion_rate['XAG/USD'] = data.loc['XAG/USD']['Price']
except KeyError:
    print("No XAG/USD - is this a US account?!")

# Add missing currencies as cross currency via USD
conversion_rate.update({currency+'/'+ACCOUNT_CURRENCY: conversion_rate[currency+'/USD'] * conversion_rate['USD/'+ACCOUNT_CURRENCY]
                        for currency in
                            (currency for currency in data['currency']
                             if currency+'/'+ACCOUNT_CURRENCY not in conversion_rate)})

# Add additional columns, calculate PipCost based on conversion rates
data['PIP'] = 10.**data['pipLocation']
data['PIPCost'] = data['PIP'] * (data['currency']+'/'+ACCOUNT_CURRENCY).map(conversion_rate)
data['Leverage'] = (1/data['marginRate']).astype(int)
data['MarginCost'], data['Commission'] = 0, 0
data['Symbol'] = data.index

# Adjust for weekend rate inflation on Wednesdays/Fridays if necessary
# data[['financing.longRate', 'financing.shortRate']] = data[['financing.longRate', 'financing.shortRate']] / 4

# Calculate RollLong and RollShort in account currency, for the quantities used by Zorro
financing_pos_size = defaultdict(lambda: 1, {'CURRENCY': 10000})    # Everything but CURRENCY defaults to 1
data['RollLong'] = data['financing.longRate']/365 * \
                   data['Price'] * \
                   data['type'].map(financing_pos_size) * \
                  (data['currency']+'/'+ACCOUNT_CURRENCY).map(conversion_rate)
data['RollShort'] = data['financing.shortRate']/365 * \
                    data['Price'] * \
                    data['type'].map(financing_pos_size) * \
                   (data['currency']+'/'+ACCOUNT_CURRENCY).map(conversion_rate)

EXPORT_COLUMNS = ['Price', 'Spread', 'RollLong', 'RollShort',
                  'PIP', 'PIPCost', 'MarginCost', 'Leverage',
                  'LotAmount', 'Commission', 'Symbol']
DISPLAY_COLUMNS = ['type', 'displayName', 'Price', 'Spread',
                   'financing.longRate', 'financing.shortRate',
                   'RollLong', 'RollShort',
                   'PIP', 'PIPCost', 'MarginCost', 'Leverage',
                   'LotAmount', 'Commission']
data.sort_values(['type', 'Name'], inplace=True)

# Write CSV with group headers
data[EXPORT_COLUMNS].head(0).to_csv(OUTPUT_FILENAME)
max((open(OUTPUT_FILENAME, 'a').write(f"### {data_type}\n"),
          data.loc[data_group, EXPORT_COLUMNS].to_csv(OUTPUT_FILENAME, float_format='%g', header=False, mode='a'))
    for data_type, data_group in data.groupby('type').groups.items())

print(data[DISPLAY_COLUMNS].to_string())

Re: Oanda instruments [Re: kalmar] #480390
06/03/20 21:22
06/03/20 21:22
Joined: May 2020
Posts: 27
Germany
M
Morris Offline
Newbie
Morris  Offline
Newbie
M

Joined: May 2020
Posts: 27
Germany
Hi DdlV,

Very useful -- thanks! (By the way, in order to check for the existence of XAG/USD, instead of try/except, you could use: if 'XAG/USD' in data.index ... -- but I realize this is not a Python forum...)

And kalmar, thanks for the pointer to finnhub.io. I look forward to checking it out!

@Eric: The code above is a Python script, so it won't run in Zorro. But it makes it easier to create the Zorro assets file for Oanda. To run it, just download and install Python and a few dependent packages.

Re: Oanda instruments [Re: kalmar] #480394
06/04/20 00:09
06/04/20 00:09
Joined: Jun 2013
Posts: 1,609
D
DdlV Offline
Serious User
DdlV  Offline
Serious User
D

Joined: Jun 2013
Posts: 1,609
Thanks, Morris, for the Python mini-lesson!

I guess one could also use try/except for the initial account access to print a nice error message rather than just having an error dump if it's a live account... Wouldn't want the code to just fix itself and forge ahead on a live account, though - bad precedent!

Regards.

Page 1 of 2 1 2

Moderated by  Petra 

Gamestudio download | chip programmers | Zorro platform | shop | Data Protection Policy

oP group Germany GmbH | Birkenstr. 25-27 | 63549 Ronneburg / Germany | info (at) opgroup.de

Powered by UBB.threads™ PHP Forum Software 7.7.1