1.79 - Spread and marketVal() in test mode

Am I doing something wrong or is the spread in test mode with marketVal() not propperly supported?


produces following log:

while the corresponding .t1 (in ZHistoryEditor, downloaded from FXCM with assetHistory(Asset,0)):

The spread of that bar was just 0. This can happen in historical data.

Spread is only stored for bars, not for ticks. So it won't change tick by tick. This is on our list for the next update - spreads will then be stored by tick.

This is exactly what one expects if he read in the manual:

I would laugh about if I didn't waste much time searching for errors in my code.

Will plugins be needing to support a new tick struct? Or will Zorro simply interpret the T1 data differently?

The .t1 files contains positive (ask) and negative values (bid). The ask can be accessed with priceClose(). If there was a chance to access the bid one could calculate the spread himself I think.

Plugins need not be changed, it is an internal Zorro function. The .t1 file is not directly stored, so you have no direct access to the bid. You could load it in parallel in a dataset and evaluate it, but better wait for the Zorro implementation. It won't take long.

Will there be a notice in What's new if it is realized?

Yes, it will.

Great!

BTW: I tried to work around using a BarPeriod of 1./60 and learned that it is useless setting it lower than one hour concerning to the spread. But Zorro reacted generally very angry to this attempt, it mixes up times (time of the end of the trade is before its start and not even of the bar period) and also the 10,000-times-higher-due-to-the-Forex-multiplicator problem appeared again (1.79.8):

Maybe those BarPeriods are not for broker arbitrage scripts?

Possibly, but I suppose that the problem is caused by many empty bars with no quotes inside. Can you contact Support with that script? They'll look into it.

Done - this happens also with the original 'Simple broker arbitrage example'.

The support's answer was that it will be fixed in next week's update.

Spread still changes according to BarPeriod, not per tick (1.80.4) or shouldn't this not being implemented yet?

Additionally there are mysterious time hoppings during tick function like:

The script is the same as at the begin of this thread, added BarPeriod = 1.

marketVal change per tick was implemented only recently.

Waited till weekend and with 1.80.6 it's really confusing because Spread=marketVal() now contains a price and it seems to be priceClose(-1) that means peeking one tick into future:

You're right. There are 2 bugs, one with the spread and one with the minute display. The minute bug is a Windows bug and can only be partially fixed, but the spread bug should be fixed in the latest version, 1.80.7

I can confirm that now there is a variable spread that changes from tick to tick. But anyhow there seems to be a problem anywhere. If I trade the simple broker arbitrage script in EUR/USD on 2 demo accounts over one week with a threshold factor of 2.5 there are 16 trades, if I test it with Spread=marketVal() over the same week there are 514 trades. I will verify it again next week and explicitely record and store the ticks of the accounts used and if those differences reappeared I'll come back with them in the broker arbitrage thread.

I hope this does not mean that one or both brokers have higher spread live than in their history. Please let me know the outcome of the test.

This is one fact I try to explore but so far this can only be concerned to FXCM, their history I downloaded in the normal way with assetHistory(Asset,0) but this week I additionaly store their ticks with the function you gave me in the "Multiple broker connection" thread (recording live) to verify any differences that might result from history creation source/way. One other broker is ActiveTrades and their history I only store while recording live because I can download tick prices via MT4 only the last 12 hours or something. Since this week I included Oanda in the test and their tick history should be downloadable with assetHistory with some restrictions (listed in the manual, I didn't try myself so far) so I can compare them with live results. I will report when the tests have finished.

So far my test results with the "simple broker arbitrage example" with a threshold factor of 2.5 extended by some plot functions and Zorro 1.80.8:

1. TEST with history FXCM from assetHistory / ActiveTrades from storing (recording) within the tick function on demo account:

Test BA_EURUSD (TICKS)

Bar period: 1 min (avg 1 min)
Test period: 2018-04-02..2018-04-06 (6955 bars)
Lookback period: 0 bars (0 minutes)
Simulation mode: Realistic ticks (slippage 5.0 sec)
Spread: marketVal()

Number of trades: 580 (43474/year, 837/week, 173/day)


SpreadA is from ActiveTrades, SpreadB from FXCM


2. TEST with history both FXCM and ActiveTrades from storing (recording) within the tick function on demo account:

Test BA_EURUSD (TICKS)

Bar period: 1 min (avg 1 min)
Test period: 2018-04-02..2018-04-06 (6955 bars)
Lookback period: 0 bars (0 minutes)
Simulation mode: Realistic ticks (slippage 5.0 sec)
Spread: marketVal()

Number of trades: 8 (600/year, 12/week, 3/day)




3. TRADE (Demo Accounts) with FXCM / ActiveTrades:

Bar period: 1 min (avg 1 min)
Trade period: 2018-04-02..2018-04-06
Spread: Live spread from broker

Number of trades: 2 (167/year, 4/week, 1/day)



For the only difference between 1 and 2 is the source of the FXCM prices it can be suspected that the spreads in their history loaded with assetHistory is another than stored within a tick function. But this does not explain all because although the spread from ActiveTrades is directly stored in the tick function it differs strongly between test and trade scenario.

I don't know if these differences can be avoided in general or if they are of systemic ones but IMO those back test results should be regarded very carefully, they can be much too optimistic (the test under point 1 has a PF of 2.34 (2.07) and a pairwise win rate of 82%). Would be nice if this could be realized.

I am curious if the Realistic ticks mode indeed has a slippage of 5sec.
5 sec seem too many, 1 sec is probably closer to reality...

Good point. I don't think there is any slippage involved. If I read the log right it seems the trades are opened the or next second the threshold's break was found:


Slippage and Fill have default values, so there should be 5 sec slippage as you mentioned, maybe there is another treatment when trading on .t1?

Hi Sphine, It seems that Case 3 is the most realistic. However, it has only 4 trades per week. I am wondering if indices e.g. Ger30, SPX500.. would be a better assets for the broker arbitrage strategies.

Hmm ... case 3 is not a simulation but recorded while trading on a demo account (therefore I wrote TRADE) and it reveals my actual problem: to find a simulation that matches trading circumstances. Those arbitrage effects often appear and disappear within one minute and therefore spread and timing at entry/exit are decisive.

If you look in the recorded ticks timestamps - then what's the actual time difference between signal generated and execution?

I wonder what the reason is for spread difference between Trade and Test of ActiveTrades?

When it comes to TEST: Does marketVal() return correct values? Are they in line with your own calculation?

The same questions for TRADE: does spread that is returned by the broker is in line with your calculation based on ticks?

The time differences are marginal, these are the corresponding ticks from AT:

2018.04.02 11:58:33.441 1,23424
2018.04.02 13:24:36.848 1,23296
2018.04.02 13:47:54.982 1,23185

I don't know if marketVal() return the right values, if I regard the ticks from above again and verify them against the last bid price, I calculate the following:

2018.04.02 11:58:33.441 1,23424
2018.04.02 11:58:33.441 -1,23419
(2018.04.02 11:58:32.988 1,23425)
(2018.04.02 11:58:32.988 -1,23419)
=> calculated spread: 0.00005, spread by marketVal(): 0.00000

2018.04.02 13:24:36.848 1,23296
2018.04.02 13:24:36.848 -1,23291
(2018.04.02 13:24:35.364 -1,23291)
=> calculated spread: 0.00005, spread by marketVal(): 0.00000

2018.04.02 13:47:54.982 1,23185
2018.04.02 13:47:54.982 -1,2318
(2018.04.02 13:47:54.482 1,23186)
(2018.04.02 13:47:54.482 -1,23181)
=> calculated spread: 0.00005 (0.00004), spread by marketVal(): 0.00000

Ticks in parentheses are the ticks before because jcl meant the bid price before could be significant, but in no case I come to the values marketVal() returns. There can be 2 reasons: I calculate wrong or marketVal() calculates wrong.

Hmm, it should be easy as 1,2,3. It looks like a bug. Have you written to support about marketVal()? How they explain zeros? Do you use the latest beta version?

Most of the info transfer between support and me concerning the marketVal() problem is documented here in the thread (it even started with a description of), therefore it's headlined with '1.79 Spread and marketVal() in test mode". 1.79 is just a bit outdated but it concerns to the actual beta. The latest results above (and the test of last week where the pics are from) are from 1.80.8/1.81.0.

Sphin, if I have problems with Zorro, I write to Support. Usually, they confirm that it's a bug or provide explanation if it was designed so. In case of marketVal() I see a problem. So the next step is to contact Support. They either confirm a bug or explain this behaviour. If you have already contacted the Support, then what was their response?

If I have problems with Zorro, esp. with a new feature and with a beta version (both is true in this case) I report it here in the beta area of the forum. I'm quite sure it will be read and if support needs more info (code, prices) they will apply for it as it has been already happened during this thread and/or the other one of mine.

I see and it's up to you. Personally, I find it more effective to write directly to Support with such kind of issues. Anyways, keep us posted and good luck!

Writing to Support is a bit faster, as I do not read the forum any day. But yes, I herewith apply for the code and data. We'd like to check why the marketval is zero.

I sent it to support. This might bring an explanation for different FXCM prices but concerning the main problem, the difference of both test scenarios to trading on demo accounts, I can't point to more that there is obviously a difference, so that testing the strategy has only little to do with trading it. But I fear I cannot provide any idea why it is as it is.

Such latency- and liquidity-sensitive strategies cannot be properly evaluated in the usual way, without "level 2" data, nor on a demo account.

You have to trade it live with small(but somewhat realistic) amounts to gauge actual fills, compare/calibrate it to you demo-account fills and/or testing results and then decide...

This is what I meant with 'systemic ones', I also fear that it might be difficult to impossible to get a realistic simulation.
BTW: I have noticed that Zorro nearly always crashes when the stop button is clicked while trading broker arbitrages. Maybe someone can look at this sometime.

Support wrote to me that the sequence of storing bid and ask in the code for recording ticks was wrong. So I will try it with changed settings again and be back with new results.

The new way recording ticks improved the test results so they seem to be more realistic but the differences between test and live runs are still there and although there are meanwhile similar (but not identical) test results using recorded and downloaded FXCM prices their differences to 'reality' are significant, they are even more significant if using real accounts instead of demos. Probably because real accounts behave more nervous esp. in shorter time frames. So I finish my efforts to simulate broker arbitrage, it is too much time consuming with only little use and turn my attention towards live trading.

Could you please share ticks recording script?