New Zorro version 2.40.9

Posted By: jcl

New Zorro version 2.40.9 - 07/07/21 12:25

Zorro 2.40 is now available:

Download: https://opserver.de/down/Zorro_240.exe

New features: contour charts, a reality check template, more cryptocurrency exchange plugins. and a lot other new features. The full list can be found on https://manual.zorro-project.com/new.htm.

This release candidate will become the official release when no bugs are found in the next time. Please test everything and report any issues here! As usual, if you're the first one to find a serious bug of this release candidate, you can get a free Zorro S subscription or support extension.
Posted By: dpn

Re: New Zorro version 2.40 - 07/19/21 17:29

Hello JCL,

I'm having problem with the updated MT5 bridge.

starting with Z7, I noted the rejection of all the orders (error 075), even changing all the possible setup in Z.ini;

I tryed to run the tradetest: no one order sent by the script is accepted by both my two different brokers;

orders sent directly by MT5 are accepted.

Please note that using the same brokers accounts, the previous zorro version is running without problem.
Posted By: jcl

Re: New Zorro version 2.40 - 07/20/21 07:05

What broker, demo or real account, and why are they rejected? You can see that in the experts log.
Posted By: jcl

Re: New Zorro version 2.40 - 07/20/21 11:44

New version 2.40.1, same URL.

Changes: New FXCM plugin and various small fixes and modifications.
Posted By: dpn

Re: New Zorro version 2.40 - 07/20/21 18:41

Hello JCL,
I have downloaded the latest version 2.40.1
Installed it on a brand new directory
I've set verbose=7 in Z.ini
Started the software as free Zorro
Same results
Tested with two brokers: Roboforex demo account leverage 500 and XM demo account with leverage 888
Orders open/closed by means of MT5 are accepted
Orders requested by tradetest.c are rejected, no more explanation on the MT5 expert tab
ref. to the annexed screenshots



Attached picture roboforex 500.jpg
Attached picture XM888.jpg
Posted By: jcl

Re: New Zorro version 2.40 - 07/21/21 07:46

From the screenshots it appears as if you have not updated the bridge. Are you sure you updated it _after_installing 2.40.1?
Posted By: dpn

Re: New Zorro version 2.40 - 07/21/21 08:23

Honestly, I have not updated the bridge, thinking it did not change since version 2.40.0.
I will try to update it as soon as possible and will keep you informed.
Thank you JCL
Posted By: dpn

Re: New Zorro version 2.40 - 07/21/21 19:39

Originally Posted by dpn
Honestly, I have not updated the bridge, thinking it did not change since version 2.40.0.
I will try to update it as soon as possible and will keep you informed.
Thank you JCL


Hi JCL,
MT5 bridge updated: All is OK, now.
Thanks.
Posted By: Grat

Re: New Zorro version 2.40.1 - 07/26/21 18:32

I open lot of order ( for the test ):

Trades 26746 Win 45.7% Avg -2.5p Bars 32

and I get in the log file 3 times the same ID order:
[643: Wed 20-04-01 00:00] (1.10268)
[EUR/USD::S64307] Short 1@1.10262 Risk 20 px at 00:00:00

[675: Wed 20-04-01 08:00] 0000 +172 32/0 (1.09605)
[EUR/USD::S64307] Expired 1@1.09564: +6.86 at 08:00:00

[10643: Thu 20-08-27 19:15] -3709 -34.47 4517/5483 (1.18247)
[EUR/USD::S64307] Short 1@1.18244 Risk 20 px at 19:15:00

[10675: Fri 20-08-28 03:15] -3737 -60.47 4519/5513 (1.18406)
[EUR/USD::S64307] Expired 1@1.18416: -1.85 at 03:15:00

[20643: Wed 21-01-27 16:15] -5844 +157 8992/11008 (1.20726)
[EUR/USD::S64307] Short 1@1.20733 Risk 20 px at 16:15:00

[20675: Thu 21-01-28 00:15] -5791 -1.18 9008/11024 (1.21044)
[EUR/USD::S64307] Expired 1@1.21031: -3.11 at 00:15:00


There's probably a counter inside that's overflowing and going round and round.
Posted By: CpOo

Re: New Zorro version 2.40.1 - 07/28/21 08:28

Hi,

I have some issues with the strategy report in last beta 2.41.1b

Same script running in 2.41b and in 2.38.5b:


Simulation mode Realistic (null)(slippage 0.0 sec) -> 2.41b
Simulation mode Realistic (slippage 5.0 sec) -> 2.38.5b

Sharpe ratio 0.00 (Sortino -1.#J) -> 2.41b
Sharpe ratio 1.51 (Sortino 1.59) -> 2.38.5b

R2 coefficient 1.000 -> 2.41b
R2 coefficient 0.394 -> 2.38.5b

Ulcer index 0.0% -> 2.41b
Ulcer index 15.1% -> 2.38.5b
Posted By: jcl

Re: New Zorro version 2.40.1 - 07/28/21 10:45

If you get different trades from different versions, first check the log. Determine which trades are different and how. You will then normally quickly see the reason. If something is unclear, post the details here.
Posted By: ozgur

Re: New Zorro version 2.40.1 - 07/28/21 16:17

I think Dataset functions are behaving differently compared to version 2.35. Is this expected?

Code:
Code
void run()
{
	
	if (is(INITRUN))
	{
		
		int handleID = 1;
		
		int len = dataParse(handleID, "ssss,i,f,f,f,f,f,f,f,f,f,f,f,f,f,f,f,f,i,i,i,i,i,i,i,i,i,i,i,i,i,i,i,i,i", "History\\Test.csv"); // Asset name 15 (4 * s - 1) character max 
		
		int i, j;
		
		for(i = 0; i < len; i++) {
			
			printf("Row %i: %s %i %i \n", i, dataStr(handleID, i, 1), dataInt(handleID, i, 5), dataInt(handleID, i, 38));
			
			for(j = 0; j < 8; j++) {
				
				printf("%f %f %i %i \n", dataVar(handleID, i, 6 + j), dataVar(handleID, i, 14 + j), dataInt(handleID, i, 22 + j), dataInt(handleID, i, 30 + j));
				
			}
			
		}
		
		// free data memory
		dataNew(handleID, 0, 0);
		
		printf("\n\n");
		
	}
}


Data: (Test.csv)
Code
Name,Apple,Orange1,Orange2,Orange3,Orange4,Orange5,Orange6,Orange7,Orange8,Melon1,Melon2,Melon3,Melon4,Melon5,Melon6,Melon7,Melon8,Grape1,Grape2,Grape3,Grape4,Grape5,Grape6,Grape7,Grape8,Plum1,Plum2,Plum3,Plum4,Plum5,Plum6,Plum7,Plum8,Berry
#AUD/CAD,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,
#AUD/CHF,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,
#AUD/JPY,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,
#AUD/NZD,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,
#AUD/USD,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,
#CAD/CHF,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,
#CAD/JPY,4,10,20,40,80,,,,,0.125,0.125,0.125,0.125,,,,,,,,,,,,,500,250,100,50,,,,,1
#CHF/JPY,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,
#EUR/AUD,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,
#EUR/CAD,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,
#EUR/CHF,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,
#EUR/GBP,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,
#EUR/JPY,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,
#EUR/NZD,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,
EUR/USD,4,12.5,25,50,100,,,,,0.125,0.125,0.125,0.125,,,,,4,4,4,4,,,,,500,250,100,50,,,,,1
#GBP/AUD,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,
#GBP/CAD,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,
#GBP/CHF,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,
GBP/JPY,4,10,20,40,80,,,,,0.125,0.125,0.125,0.125,,,,,8,8,8,8,,,,,500,250,100,50,,,,,1
#GBP/NZD,4,25,50,100,200,,,,,0.125,0.125,0.125,0.125,,,,,,,,,,,,,500,250,100,50,,,,,1
#GBP/USD,4,16,32,64,128,,,,,0.125,0.125,0.125,0.125,,,,,,,,,,,,,500,250,100,50,,,,,1
#NZD/CAD,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,
#NZD/CHF,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,
#NZD/JPY,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,
#NZD/USD,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,
#USD/CAD,5,10,30,50,80,130,,,,0.125,0.125,0.125,0.125,0.125,,,,,,,,,,,,500,250,100,50,25,,,,1
#USD/CHF,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,
#USD/JPY,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,


Result: (just part of it to give you an idea)

[Linked Image]
Posted By: CpOo

Re: New Zorro version 2.40.1 - 07/28/21 17:41

Originally Posted by jcl
If you get different trades from different versions, first check the log. Determine which trades are different and how. You will then normally quickly see the reason. If something is unclear, post the details here.


No, I get exactly the same trades, same profit, annual growth rate, percent winning, MAE and so on... same report but the only difference between versions are those values and the (null) value in Simulation mode.
Posted By: jcl

Re: New Zorro version 2.40.1 - 07/28/21 18:11

Hmm, that sounds indeed strange. Can you contact support@opgroup.de and send them that script? They'll look into it.

Ozgur: I believe the first field is 0, not 1. Indices start with 0. The other column numbers are probably also wrong. Can you check if it works when you fix the numbers? Datasets expect a date in field 0, so I don't know what happens when it is missing, but possibly the old version got then some phantom field and thus worked with the wrong numbers.
Posted By: CpOo

Re: New Zorro version 2.40.1 - 07/28/21 18:42

Originally Posted by jcl
Hmm, that sounds indeed strange. Can you contact support@opgroup.de and send them that script? They'll look into it.



I am using your Alice3b.c script for testing, but I am seeing the problem with any script. I tested in 2 different computers before writing here. It only happens on Test mode. Trade mode reports are Ok.
Posted By: jcl

Re: New Zorro version 2.40.1 - 07/29/21 08:33

Ah, ok. We'll check that and when it happens here too, it will be fixed. Since you found a bug in a release candidate, you're entitled for a 3 months free Zorro S subscription or support period - please contact Support for claiming the reward.

Ozgur: You can also claim a free subscription. The problem was indeed the missing date field, but both versions did not react consistently when it's missing. This will be fixed so that datasets with no date are now officially supported. But you must then still change your field numbers. The first field is 0 and the next integer field is 3, since the first 3 fields are needed for the string.
Posted By: jcl

Re: New Zorro version 2.40.1 - 07/29/21 12:09

2.40.2 is now out. Changes: Sharpe ratio fixed and datasets with no date field supported.
Posted By: SBGuy

Re: New Zorro version 2.40.2 - 08/01/21 16:27

My 2-minute bar options strategy surfaced a problem on 2.40.2

This is the code I use to load the options chain for the upcoming bar inside tock, when in Test mode.

The month() call is used to specify the T8 file name.

As you can see from the attached screenshot, month() stays at 1 when the actual month changed to February. This continues for the entire simulation period, LOOKBACK or not, resulting in no trades.

This code works fine in 2.35.6 and previous version going back at least couple of years.

Looks like month() is acting weird inside of tock(), or maybe elsewhere too.

Code

if (!Live)	{
		
		printf("\n[%d] TOCK -- [%s] Loading chain... ",utcHHMM, Asset);
		if(month()!=month(1) or Init) {		
			sprintf(OptionFile,"%s\\%s_%d%02d_m1.t8", HistFolder, Asset, year(),month());
			dataLoad(1,OptionFile,9);	
		}
		
		if (!Init)	{
			int Chain = contractUpdate(Asset,1,PUT|CALL); 
			if(Live || DEBUG) printf("\n[%d] TOCK -- [%s] %d contracts loaded...\n\n",utcHHMM, OptionFile, Chain);	
		}
	}


Attached File
tock-chain-load.PNG  (69 downloads)
Posted By: SBGuy

Re: New Zorro version 2.40.2 - 08/01/21 17:09

Another anomaly.

Assets with not enough historical data used to throw a "Warning 055: GRUB first yyy bars missing" message during Init only.

Now, v 2.40.2, throws up "Error 047: GRUB not enough ticks." during Init and every..... single.... bar. Even with a Verbose=1.

As a result, the large amount of error messages printed to the console window makes the simulation time unnecessarily long.

I could ignore(47), but that's probably not a good idea.

I suggest we go back to the Warning 055message during Init, like before.

Thanks!
Posted By: kzhao

Re: New Zorro version 2.40.2 - 08/01/21 20:03

I tested a couple of my strategies on 2.40.2. The SR in the result is always 0, while in 2.35 the SR has positive values.
Posted By: jcl

Re: New Zorro version 2.40.2 - 08/02/21 08:34

Some comments:

- Wrong month: Maybe it's some setting of your script? Find out with a simple script like this:

void tock() { printf("\n%02d",month()); }
void run() { asset("EUR/USD"); }

Then add your settings at the begin of the run function until the month becomes wrong again. Please let us know what it was.

- Too many errors 047: Yes, this will be limited to 2 or 3 errors.

- SR is 0 with 2.40.2: Please post or send that script.
Posted By: SBGuy

Re: New Zorro version 2.40.2 - 08/03/21 00:49

hi jcl,

It looks like the problem is not the month() function, but rather the month(1) offset at the 1st bar of a new month.

As you can see from the log snippet below, the first bar on 21-02-01 shows: month(0)=2 and month(1)=2.

You agree that it should be: month(0)=2 and month(1)=1?

Code

function tock()
....
	printf("\n[%d] TOCK -- [%s] month()=%d  month(1)=%d... ",utcHHMM, Asset,month(),month(1));
	if(month()!=month(1) or Init) {		
		sprintf(OptionFile,"%s\\%s_%d%02d_m1.t8", HistFolder, Asset, year(),month());
		printf("T8: %s\n",OptionFile);
	}


..................Log entry................

[974: Fri 21-01-29 20:58] 3730.72/3734.22\3727.72/3727.72 -0.01
[2058] TOCK -- [SPX] month(0)=1 month(1)=1...

[975: Mon 21-02-01 14:30] 3727.72/3727.72\3703.61/3707.86 -0.01
[1430] TOCK -- [SPX] month(0)=2 month(1)=2...
Posted By: jcl

Re: New Zorro version 2.40.2 - 08/03/21 09:12

Afraid not, but I now see the reason of the problem.

Inside a tick or tock function, month(0) is the current time. month(1) is the end of the previous bar. Depending on when the function runs, both can be identical. So your comparison is not good, even if it may have worked sometimes. For correctly finding a month change, either compare month(0) != month(1) in the run function, or compare month(0) in the tock function with month(0) of the previous tock.
Posted By: jcl

Re: New Zorro version 2.40.2 - 08/03/21 09:37

2.40.3 is now out. Changes: Multiple 047 error messages suppressed, and fixes for the other reported issues.
Posted By: SBGuy

Re: New Zorro version 2.40.2 - 08/03/21 12:32

hi jcl,

I hear what you're saying. But I still believe there is a problem in 2.40 when tock was moved to execute after tick, which makes sense conceptually.

Take a look at these log clippings and you'll see the problem.

In v2.35, tock is executed before run(), and tock() has the same time stamp and BarNum as the upcoming run() , which make sense.

In v2.40, tock is executed before run(), and tock() has the new BarNum, BUT..... old timestamp of the previous bar.

This probably causes month(1) to equal month(0). I also did do as you suggested and check for month change in run(). But because of this problem, I am only able to detect a month change in bar 2 of the new month.

FYI, this the formula I used for the timestamp printed in the log.

int utcHHMM = hour()*100+minute();

Code
..................... v 2.35 and before .........................

[1430] Bar[975] TOCK -- [SPX] 
[975: Mon 21-02-01 14:30] 3727.72/3727.72\3703.61/3707.86 -0.01
[1430] Bar[975] run() -- price=3707.86

[1432] Bar[976] TOCK -- [SPX] 
[976: Mon 21-02-01 14:32] 3751.48/3753.72\3751.48/3753.72 -0.01
[1432] Bar[976] run() -- price=3753.72

................... v2.40 ............................................

[974: Fri 21-01-29 20:58] 3730.72/3734.22\3727.72/3727.72 -0.01
[2058] Bar[974] run() -- price=3727.72

[2058] Bar[975] TOCK -- [SPX]   <------------------------ bad time stamp for Bar 975. it should be 1430
[975: Mon 21-02-01 14:30] 3727.72/3727.72\3703.61/3707.86 -0.01
[1430] Bar[975] run() -- price=3707.86

[1430] Bar[976] TOCK -- [SPX] <--------------------------- bar time stamp for Bar 976.  it should be 1432
[976: Mon 21-02-01 14:32] 3751.48/3753.72\3751.48/3753.72 -0.01
[1432] Bar[976] run() -- price=3753.72


Posted By: jcl

Re: New Zorro version 2.40.2 - 08/04/21 07:09

The tock timestamp is _not_ the end of the next bar. It is simply the current time. In 2.40 tock is run by the internal state machine in the backtest, so that tock timestamps are always synchronized to the ticks. In 2.35, tock timestamps were unsynchronized.

Whether to detect month changes in run or in tock depends on what for you're using it. If your script must react immediately on a new month, even in the middle of a bar, then you must check the change of month(0) inside tock or tick. That's easy with a static variable.
Posted By: CpOo

Re: New Zorro version 2.40.2 - 08/04/21 09:01

Originally Posted by jcl
Ah, ok. We'll check that and when it happens here too, it will be fixed. Since you found a bug in a release candidate, you're entitled for a 3 months free Zorro S subscription or support period - please contact Support for claiming the reward.


Thanks!
Posted By: SBGuy

Re: New Zorro version 2.40.2 - 08/04/21 15:26

hi jcl,

My understanding in 2.40 is that execution order is now: tick, TOCK, TMF, run.... instead of TOCK, tick, TMF, run as in 2.35.

My observation from this log snippet below is that a contractUpdate() in Bar 976 tock, will load the options chain from time 1430, which is Bar 975. The TMF that executes before run will used the options chain from 1430. This problem may not be that big using 2 minutes bars, but will be a big problem longer bars.

So... in short, my original month detection problem has evolved to a potential tick/tock/run timing problem.

See the log below.

In v2.35 all the bars have the same time stamp, i.e. tick, tock, run associated with Bar 976 have a timestamp of 1432, which makes sense.
In v2.40 tick, tock in Bar 976 have a time of 1430, and run has a time of 1432... doesn't look right to me.

Thanks for looking closer at this.

Code

................... v2.35 ......................... TOCK, tick, TMF, run

[1430] Bar[975] TOCK -- [SPX] LastMonth[1] RunMonth[1] NewMonth[0]... USE NEW T8: C:\Zorro\History\SPX-Options\\SPX_202102_m1.t8
[1430] Bar[975] tick -- [SPX] LastMonth[1] RunMonth[1] NewMonth[0]... 
[975: Mon 21-02-01 14:30] 3727.72/3727.72\3703.61/3707.86 -0.01
[1430] Bar[975] run() -- price=3707.86

[1432] Bar[976] TOCK -- [SPX] LastMonth[1] RunMonth[2] NewMonth[1]... USE NEW T8: C:\Zorro\History\SPX-Options\\SPX_202102_m1.t8
[1432] Bar[976] tick -- [SPX] LastMonth[1] RunMonth[2] NewMonth[1]... 
[976: Mon 21-02-01 14:32] 3751.48/3753.72\3751.48/3753.72 -0.01
[1432] Bar[976] run() -- price=3753.72

.................. v2.40 ......................... tick, TOCK, TMF, run

[2058] Bar[975] tick -- [SPX] LastMonth[1] RunMonth[1] NewMonth[0]... 
[2058] Bar[975] TOCK -- [SPX] LastMonth[1] RunMonth[1] NewMonth[0]... 
[975: Mon 21-02-01 14:30] 3727.72/3727.72\3703.61/3707.86 -0.01
[1430] Bar[975] run() -- price=3707.86

[1430] Bar[976] tick -- [SPX] LastMonth[1] RunMonth[2] NewMonth[1]... 
[1430] Bar[976] TOCK -- [SPX] LastMonth[1] RunMonth[2] NewMonth[1]... USE NEW T8: C:\Zorro\History\SPX-Options\\SPX_202102_m1.t8
[976: Mon 21-02-01 14:32] 3751.48/3753.72\3751.48/3753.72 -0.01
[1432] Bar[976] run() -- price=3753.72



The test script

Code
#include <stdio.h>
#include <default.c>

#define TICKER			"SPX"

string HistFolder[200];
string OptionFile[200];

bool NewMonth;
int LastMonth, RunMonth;

void main()
{	
	sprintf(HistFolder,"C:\\Zorro\\History\\%s-Options\\",TICKER);		
}


function tick()	
{
	int utcHHMM = hour()*100+minute();
	printf("\n[%d] Bar[%d] tick -- [%s] LastMonth[%d] RunMonth[%d] NewMonth[%d]... ",utcHHMM, Bar, Asset,LastMonth, RunMonth, NewMonth);	
}

function tock()	
{
	int myHour = lhour(ET);
	int myMins = minute();
	int HHMM = myHour*100+myMins;
	
	int utcHHMM = hour()*100+minute();

	printf("\n[%d] Bar[%d] TOCK -- [%s] LastMonth[%d] RunMonth[%d] NewMonth[%d]... ",utcHHMM, Bar, Asset,LastMonth, RunMonth, NewMonth);
	if(month(0)!=LastMonth) {		
		sprintf(OptionFile,"%s\\%s_%d%02d_m1.t8", HistFolder, Asset, year(),month());
		printf("USE NEW T8: %s\n",OptionFile);
		//contractUpdate().....
	}
}


void run() 
{
	NewMonth = month(0)!=month(1);
	RunMonth = month(0);
	LastMonth = month(1);
	
	int utcHHMM = hour()*100+minute();

	Verbose = 3;	//7|DIAG;

	StartDate = 20210301;
	EndDate = 	20210303;	
		
	static char HistoryPath[100];
	strcpy(HistoryPath,strf("%s\\*_mm.t6",HistFolder));
	History = HistoryPath;		

	BarPeriod = 2;		
	LookBack = 115*30;	// 115 1-hour bar look back
	
	BarZone = ET;
	StartMarket = 0930;			//ET Time
	EndMarket = 1600;				//ET Time
	BarMode = BR_WEEKEND+BR_MARKET+BR_SLEEP;
	TickTime = 1*60*1000;
	TockTime = 1*60*1000;
	set(PRELOAD);
	set(LOGFILE);
	
	asset("SPX");
	printf("\n[%d] Bar[%d] run() -- price=%.2f\n", utcHHMM, Bar, priceClose());
	
}

Posted By: jcl

Re: New Zorro version 2.40.2 - 08/05/21 13:51

Hmm. If bar N ends with time t1, and bar N+1 ends with time t2, then I would normally expect for all the ticks and tocks of bar N+1 timestamps that start with t1 and end with t2. Won't you agree?
Posted By: SBGuy

Re: New Zorro version 2.40.2 - 08/05/21 17:09

I'm not sure I know enough about the internals of Zorro to opine on your question.

But let me ask you this in the context of my specific application, using the details of the log entries below:

If contractUpdate() is called in Bar 976 TOCK, or tick, which has a timestamp of 1430, which contract chain do you think will be loaded?
(a) the chain for time of 1430 (the tick/tock timestamp)
(b) the chain for time of 1432 (the Bar 976 time)

Code
.................. v2.40 ......................... tick, TOCK, TMF, run

[2058] Bar[975] tick -- [SPX] LastMonth[1] RunMonth[1] NewMonth[0]... 
[2058] Bar[975] TOCK -- [SPX] LastMonth[1] RunMonth[1] NewMonth[0]... 
[975: Mon 21-02-01 14:30] 3727.72/3727.72\3703.61/3707.86 -0.01
[1430] Bar[975] run() -- price=3707.86

[1430] Bar[976] tick -- [SPX] LastMonth[1] RunMonth[2] NewMonth[1]... 
[1430] Bar[976] TOCK -- [SPX] LastMonth[1] RunMonth[2] NewMonth[1]... USE NEW T8: C:\Zorro\History\SPX-Options\\SPX_202102_m1.t8
[976: Mon 21-02-01 14:32] 3751.48/3753.72\3751.48/3753.72 -0.01
[1432] Bar[976] run() -- price=3753.72
Posted By: jcl

Re: New Zorro version 2.40.2 - 08/06/21 09:55

Hopefully a. Otherwise you had a problem.
Posted By: SBGuy

Re: New Zorro version 2.40.2 - 08/06/21 15:34

Interesting. I was hoping for (b) because that's how it works in 2.35 where all the timestamps for tick/tock/run are the same.

Here's another log with priceClose(0) displayed.

As you can see, Bar 976 tick/TOCK has a timestamp of 1430, and yet the priceClose(0) of 3753.72 is actually from the end of Bar 976. This means regardless of timestamp tick/tock is getting prices at the end of Bar 976, which is 1432.

If you agree that this is the correct behavior, then I would submit that contactUpdate inside of tock should ALSO pull the contract chain at 1432, as options prices are mathematically linked to the UL price.

Imagine the BarPeriod=60, the UL price and corresponding options chain, inside tock, will be 1 hour apart!!! That's a big problem.

There was not a problem in 2.35 and prior versions.

Thanks.

Code

[2058] Bar[975] tick -- priceClose()[3707.86] [SPX] LastMonth[1] RunMonth[1] NewMonth[0]... 
[2058] Bar[975] TOCK -- priceClose()[3707.86] [SPX] LastMonth[1] RunMonth[1] NewMonth[0]... 
[975: Mon 21-02-01 14:30] 3727.72/3727.72\3703.61/3707.86 -0.01
[1430] Bar[975] run() -- price=3707.86

[1430] Bar[976] tick -- priceClose()[3753.72] [SPX] LastMonth[1] RunMonth[2] NewMonth[1]... 
[1430] Bar[976] TOCK -- priceClose()[3753.72] [SPX] LastMonth[1] RunMonth[2] NewMonth[1]... USE NEW T8: C:\Zorro\History\SPX-Options\\SPX_202102_m1.t8
[976: Mon 21-02-01 14:32] 3751.48/3753.72\3751.48/3753.72 -0.01
[1432] Bar[976] run() -- price=3753.72


Posted By: jcl

Re: New Zorro version 2.40.2 - 08/06/21 16:59

Ok, I understand the problem with the close price. Since you are not using TICKS mode, the only available close price is the close of the bar. More correct would be returning the bar open price instead. But priceClose returning an open price would be somewhat inconsistent.

We'll think about a solution.
Posted By: SBGuy

Re: New Zorro version 2.40.2 - 08/06/21 17:58

Thank you for looking into this.

Also, take a look at TMF timing as well, and the contract prices available to it.
Posted By: 7th_zorro

Re: New Zorro version 2.40.3 - 08/07/21 06:00

I update version 2.35 to 2.40.3.
I used same back test data, but process differently.
Plotting .t6 file is ok, but when backtesting it process differently.
I attach capture image, price data looks ok but bactesting result show wrong chart. also cause wrong backtesting result.

Attached picture Capture.PNG
Posted By: Petra

Re: New Zorro version 2.40.3 - 08/07/21 06:22

When plotting is ok but the backtest has strange candles then something in your script produces wrong prices. Its either a script bug or a zorro bug. Send that script and history to support. If its a zorro bug you can get a free subscription.
Posted By: 7th_zorro

Re: New Zorro version 2.40.3 - 08/07/21 06:48

Hi, Petra.
I sent the report mail to zorro support team.
Please check the result.
Thank you.
Posted By: Petra

Re: New Zorro version 2.40.3 - 08/10/21 08:21

Thanks. Turns out the reason was high volatility and too many outliers in the data, triggering the outlier detection.

If anyone has a similar problem, the fix:

Outlier = 0;

This disables outlier detection. The outlier detector will be made less sensitive for the release.

Posted By: 7th_zorro

Re: New Zorro version 2.40.3 - 08/11/21 01:14

Hi, Thank you for the solution.
The problem is now fixed.
I don't know why the sensitivity changed in new version,
but would better be careful changing something.
I was very confused why new version proceed differently when I just update to new version.

I hope you to kindly provide new subscription for me. Would you?
Posted By: Petra

Re: New Zorro version 2.40.3 - 08/11/21 11:42

Ask them by email. It's not a real bug and also mentioned in the manual so I'm not sure but asking doesnt hurt.
Posted By: CpOo

Re: New Zorro version 2.40.3 - 08/11/21 12:53

Hi,

Can confirm that in the 2.40.3 version installation the Binance Futures Plugin is missing. I understand that it's possible to copy the dll from the 2.41.1 to the 2.40.3 without any problem, right?

BTW: 2.41.1b is still not fixed with the latest reported issues, 2.40.3 is.
Posted By: jcl

Re: New Zorro version 2.40.3 - 08/12/21 08:21

I cannot confirm any missing plugins, but I can confirm that 2.41.1 beta corresponds to the old 2.40.1 release. There will be a new beta in the next days.
Posted By: jcl

Re: New Zorro version 2.40.3 - 08/13/21 09:12

New version: 2.40.4.

Changes: Default outlier detection is now less sensitive. And after many discussions, we have now moved tick and tock from the start to the end of the bar when no TICKS flag is set. This makes close price and timestamp match. It TICKS is set, or in trade mode, the behavior is as before.
Posted By: SBGuy

Re: New Zorro version 2.40.3 - 08/13/21 16:50

Hi jcl,

Thanks for the tick/tock time fix. I can confirm that tick/tock does in fact run at the end of the bar in v2.40.4

However, the tmf that execute before the run() is still running at the beginning of the bar. This difference between v2.35 and v2.40 causes meaningfully significant difference in backtest results.

In 2.40.4, the execution sequence looks like this:
tmf [timestamp = bar begin]
tick [timestamp = bar END]
tock [ timestamp = bar END]
run [timestamp = bar END]

I believe the TMF in this group should also execute at a timestamp=bar END, and with this sequence

tick [timestamp = bar END]
tock [ timestamp = bar END]
tmf [timestamp = bar END]
run [timestamp = bar END]

Basically tmf should run AFTER tick/tock, like v 2.35

Hope you agree.

Thanks.
Posted By: CpOo

Re: New Zorro version 2.40.3 - 08/14/21 10:58

Thanks for the new fixed version.

Binance Futures plugin is not missing this time and it seems that it's updated from 1.0.2.6 to 1.0.2.7, but couldn't find if there are any important changes.

BTW: May I claim a perpetual license due to my report? blush
Posted By: AndrewAMD

Re: New Zorro version 2.40.3 - 08/14/21 12:47

1.0.2.7 was just a minor bug fix.
Posted By: jcl

Re: New Zorro version 2.40.3 - 08/16/21 07:18

There is no execution sequence for TMFs. They are triggered by ticks, entry, stop, and other events, and run when that event happens. In case of ticks they run at the same time as the tick function.
Posted By: SBGuy

Re: New Zorro version 2.40.3 - 08/16/21 15:34

The TMF below executes immediately before the upcoming run(), after getting the latest tick price.

As you can see, it ran before tick and TOCK. As a result it doesn't get the options chain for time 1840 that was loaded in TOCK.

What I am suggesting is that instead of this execution sequence:
TMF, tick, TOCK, run

It should be this, just like 2.35 and before.
tick, TOCK, TMF, run

I don't think making this change will cause any internal Zorro issues at all. In fact, I think it'll make it better for all TMF events:
tick, TOCK, TMF event....

From v.2.40.4
Code

[1840] INSIDE myTMF[489449]: priceClose(0)=3816.59 
2021-03-05,Call,20210308,3810.0,3815.3,32.60,32.10,0.529,773,
ContractBid[32.10]  ContractAsk[32.60]  TradeStopLimit[27.75]

[4895: 1840] tick priceClose[3816.59]

[4895: 1840] TOCK -- [SPX] Loading chain... 
[4895: 1840] TOCK priceClose[3816.59] -- [C:\Zorro\History\SPX-Options\\SPX_202103_m1.t8] 600 contracts loaded...

[4895: Fri 21-03-05 18:40] 860 -1912 5/4 3815.25/3816.59\3815.09/3816.59 -0.01



Posted By: jcl

Re: New Zorro version 2.40.3 - 08/18/21 07:25

Check the TradeIs... flags. You can then see when and why a TMF was triggered.
Posted By: jcl

Re: New Zorro version 2.40.5 - 08/18/21 07:30

New version: 2.40.5

The bugs, a crash and a wrong contour chart, that were reported for 2.40.4 have been fixed in this version.
Posted By: SBGuy

Re: New Zorro version 2.40.5 - 08/19/21 15:46

From 2.40.5. As you can see, none of the other tmf events were triggered at 1840. Therefore it must be triggered by a normal tick.

All I am saying is that the TMF execution, regardless of event type, should run AFTER tick() tock(), like it was in v.2.35 and many version before.

The manual says "TMF runs at every tick". Therefore you must allow tick() and tock() to completely finish and provided any pre-processing that is related to that tick. In my case, that pre-processing is loading tick specific options chains, before TMF can run so that the TMF can have all the latest prices associated with that new tick.

In my example below, the UL price ($3816.59) is from time 1840 and the options price is from time 1838 (UL=$3815.3). This is wrong.

Thanks.

Code
[1840] INSIDE myTMF[489449]: priceClose(0)=3816.59 
TradeIsEntry = 0
TradeIsStop = 0
TradeIsProfit = 0
TradeIsClosed = 0
2021-03-05,Call,20210308,3810.0,3815.3,32.60,32.10,0.529,773,   <------ this option data is from the previous tick at time 1838
ContractBid[32.10]  ContractAsk[32.60]  TradeStopLimit[27.75]

[4895: 1840] tick priceClose[3816.59]

[4895: 1840] TOCK -- [SPX] Loading chain... 
[4895: 1840] TOCK priceClose[3816.59] -- [C:\Zorro\History\SPX-Options\\SPX_202103_m1.t8] 600 contracts loaded...

[4895: Fri 21-03-05 18:40] 860 -1912 5/4 3815.25/3816.59\3815.09/3816.59 -0.01SliderLots = 20

Posted By: jcl

Re: New Zorro version 2.40.5 - 08/20/21 08:32

Yes, when triggered by a price quote the TMF always runs before tick. It must react fast to the new price. Tick has lower priority. Tock is anyway not price triggered.
Posted By: SBGuy

Re: New Zorro version 2.40.5 - 08/20/21 15:23

So, it looks like I can't convince you to revert things back to 2.35 behavior.

What do you suggest I do as a work around?

My TMF must have a synchronized options chain for the underlyprice/timestamp the TMF is executing. Otherwise the result is just plain wrong.

I moved the contractUpdate() call to inside the TMF and it DID load the correct options chain, but it also caused an infinite loop on the first TMF call.

Posted By: Petra

Re: New Zorro version 2.40.5 - 08/22/21 10:09

Option chains should in live mode only be loaded once per day and only in the run function. It can take 1 hour to load an option chain. This cannot work with tock or tmf. i have also not heard of someone who had that idea.

If the tmf needs the current price, call contractprice. Infinite loops in tmfs are mostly by opening new trades and producing an infinite recursion.
Posted By: SBGuy

Re: New Zorro version 2.40.5 - 08/22/21 14:30

As said previously, this problem is for backtest only. There are no problems in live trading because contractPrice gets the most current contract prices from the broker plugin.

As you know in backtest, contractPrice called in TMF only uses price data from whatever historical contract chain was loaded in the most recent contractUpdate.

As you know, contractUpdate only loads the snippet of options chain from the large historical .t8 file, ONLY for the "current time."

My work around for this limitation in the past was to call contractUpdate in tock, when TMF runs after tock. Thereby providing the latest contractPrices to TMF.

In 2.40, you guys changed the execution order from tick, tock, TMF, run to: TMF, tick, tock, run.

As a result, the current time for TMF lags the options chain time for exactly 1 Bar, or 1 tick (if TICKS flag was applicable).

Since options price and UL price are mathematically correlated, it is wrong to have priceClose() and contractPrice() be from 2 different times.

I've been trying to explain this for the past few days, and I can't seem to be get through.

I don't know what the reason was to change the execution order to TMF, tick, tock, run after many years.

I am saying this new execution sequence is fundamentally wrong in backtest. If backtest is flawed, what's the point of live trading?

jcl said this earlier "when triggered by a price quote the TMF always runs before tick. It must react fast to the new price."

v 2.40 reacts fast to the latest UL price ONLY, but not the latest contractPrice.

Change the execution order back to tick, tock, TMF, run, and everything will be fine.... at least for backtest.

Thank you.
Posted By: jcl

Re: New Zorro version 2.40.6 - 08/23/21 07:29

I already tried to explain that the order of function calls was not changed, since previous versions had no particular order. It depended on circumstances. Only from 2.40 on there is a state machine that triggers events in a fixed order. But that does not mean that you should use that order in your script.

If a script only works with a certain order of events, I would normally consider that a bug. I would not accept such a script from a freelancer unless there is a special reason for it. If you need a certain order of function calls, have code in your script that calls these functions in that order. Update the options first and update your trades afterwards. I don't see why you would need some tick or tock for that.

If there is a problem that you cannot solve, post it. If there is really no solution with the current Zorro version, we'll implement one. But I doubt that this will be needed.
Posted By: jcl

Re: New Zorro version 2.40.7 - 08/30/21 08:18

New version: 2.40.7

A bug with cancelTrade was fixed.
Posted By: 7th_zorro

Re: New Zorro version 2.40.7 - 08/30/21 11:51

I know you are busy, but can you give me a rough schedule regarding 64bits support? Thank you.
Posted By: CpOo

Re: New Zorro version 2.40.7 - 08/31/21 18:51

Hi,

I found an important problem (IMHO) with the latest release candidate related to data download from the Broker. Not sure if the problem is with assetHistory or with the broker Plugin (or maybe it's not a problem at all but my problem or my miss understanding) but I am seeing 1 minute gap each time I download new prices. Also in trade mode. I only test this with the Binance Futures plugin.

Steps to reproduce the problem:

Script: Download.c with following config:

Code
#define VERBOSE	7+DIAG

And also I include Outlier = 0; just in case:
Code
void main()
{
...
Outlier = 0;
...

In the Download panel:

NKNUSDT
2021-2021
M1
Ask/Bid
Broker
-> Click Download

From the .diag file:

Code
!Connecting to  websocket...
!Websocket connected.
printf: 
!§s: 1 2901.173600 ms
BrokerTime: 2021-08-31  16:42:06: 2 7.525700 ms
printf:  at UTC §s
Open NKNUSDT  >Save
Open: Log\Assets.csv wb  >Hist  >File History\NKNUSDT_2021.t6  >Prices
Hist 01-01 00:00 .. 08-31 16:42: 388313
printf: 
§s: §i MB allocated
Ticks:
BrokerHistory NKNUSDT: 
HTTP: https://fapi.binance.com/fapi/v1/klines?symbol=NKNUSDT&interval=1m&startTime=1630427819000&endTime=1630428126000&limit=1500,(null),(null)5
printf: 
!§s §s to §s
printf: 
!=> §s to §s, §i ticks 5
printf: 
Load §s 
printf: §d..
Open: History\NKNUSDT_2021.t6 wb
printf:  §i min
Total: 5 207720 0
Raus
Neural
Login BinanceFutures 1->0
printf: 
Logout..


The script loads 5 bars from the broker and add them to the .t6 file. I can see them with History.c script and also with the ZHistoryEditor.

First problem is that the Broker API call returns 6 bars, and it seems that Zorro is not saving the last one. If you paste the above URL (https://fapi.binance.com/fapi/v1/kl...amp;endTime=1630428126000&limit=1500) where:

startTime is Tue Aug 31 2021 16:36:59
endTime is Tue Aug 31 2021 16:42:06

You can see the 6 prices from Binance:

Code
[[1630427820000,"0.44780","0.44780","0.44600","0.44665","89850",1630427879999,"40155.24761",142,"21596","9652.74538","0"],
[1630427880000,"0.44646","0.44666","0.44454","0.44474","92428",1630427939999,"41153.78206",173,"10736","4787.09330","0"],
[1630427940000,"0.44496","0.44640","0.44496","0.44601","103252",1630427999999,"45995.73933",121,"87631","39028.90056","0"],
[1630428000000,"0.44628","0.44633","0.44580","0.44587","11299",1630428059999,"5039.44539",51,"5783","2579.46923","0"],
[1630428060000,"0.44585","0.44607","0.44574","0.44607","10934",1630428119999,"4875.16577",38,"5078","2264.55739","0"],
[1630428120000,"0.44586","0.44593","0.44570","0.44584","6215",1630428179999,"2770.86145",24,"4605","2053.17810","0"]]


First bar is Tue Aug 31 2021 16:37:00
Last bar is Tue Aug 31 2021 16:42:00

But when I check the .t6 file with the ZHistoryEditor I don't see the last bar and prices finish at 16:41 with this bar from binance:

Code
[1630428060000,"0.44585","0.44607","0.44574","0.44607","10934",1630428119999,"4875.16577",38,"5078","2264.55739","0"],


I repeated the same steps 2 minutes later:

Code
!Connecting to  websocket...
!Websocket connected.
printf: 
!§s: 1 2740.935300 ms
BrokerTime: 2021-08-31  16:44:05: 2 9.992100 ms
printf:  at UTC §s
Open NKNUSDT  >Save
Open: Log\Assets.csv wb  >Hist  >File History\NKNUSDT_2021.t6  >Prices
Hist 01-01 00:00 .. 08-31 16:44: 388315
printf: 
§s: §i MB allocated
Ticks:
BrokerHistory NKNUSDT: 
HTTP: https://fapi.binance.com/fapi/v1/klines?symbol=NKNUSDT&interval=1m&startTime=1630428179000&endTime=1630428245000&limit=1500,(null),(null)1
printf: 
!§s §s to §s
printf: 
!=> §s to §s, §i ticks 1
printf: 
Load §s 
printf: §d..
Open: History\NKNUSDT_2021.t6 wb
printf:  §i min
Total: 1 207725 0
Raus
Neural
Login BinanceFutures 1->0
printf: 
Logout..


Zorro loads and save 1 bar (only the first one), but Binance returns 2 bars:

Code
[[1630428180000,"0.44586","0.44606","0.44575","0.44603","12522",1630428239999,"5583.24085",41,"7296","3253.43500","0"],
[1630428240000,"0.44608","0.44674","0.44590","0.44672","26771",1630428299999,"11943.50777",59,"8209","3663.38510","0"]]


The result is a 1 bar gap between each Download call.

I attach a ZHistoryEditor capture (gaps1.jpg). Please note the gap at 16:42, between both Downloads described above. Bar at 16:42 is missing (from the first download call) and also last 16:44 bar is missing.

If I try to Download again, no matter when, Download will start at 16:45 as Zorro thinks that 16:44 is loaded, but it's not. See capture2.jpg.

I also try to download fresh data with assetHistory and OVERRIDE but with same result. First 1500 bars are loaded from Binance, but only 1499 are saved and if I repeat again new ones start 1 minute after the last saved, with the gap in between.

Last example. Log from Trade mode (_real.log) from other Asset:

Code
...
[3304: Tue 21-08-31 16:01c] 47439.56/47450.00\47410.28/47425.61 -0.01
[3305: Tue 21-08-31 16:02c] 47425.61/47472.37\47361.71/47396.98 -0.01
[3306: Tue 21-08-31 16:03c] 47396.98/47464.10\47396.90/47464.10 -0.01
[3307: Tue 21-08-31 16:05c] 47479.74/47547.73\47479.74/47537.50 -0.01
[3308: Tue 21-08-31 16:06c] 47537.50/47550.00\47494.00/47500.55 -0.01
[3309: Tue 21-08-31 16:07c] 47500.55/47501.84\47432.32/47434.01 -0.01
[3310: Tue 21-08-31 16:08c] 47434.02/47452.39\47385.72/47411.86 -0.01
...

End of lookback period
...


Price at 16:04 is missing. I downloaded prices at 16:04 and then I download again few minutes later.

Finally... I also try to download from the Binance API at the same time that I call the API, just to double-check that Binance had the prices at the download time, and the prices were there and can't figure out why Zorro is not saving the last one.

Attached picture gaps1.jpg
Attached picture capture2.jpg
Posted By: jcl

Re: New Zorro version 2.40.7 - 09/01/21 17:28

Hmm, we'll look into that. Maybe some issue with the Binance plugin.
Posted By: jcl

Re: New Zorro version 2.40.9 - 09/03/21 08:46

Version 2.40.8: History fix for the Binance Futures plugin, and a possible gap after the lookback history is now filled with a single bar, not with multiple bars.

2.40.9: Fix for finding contracts with less than 1 day expiration.
Posted By: jcl

Re: New Zorro version 2.40.9 - 09/13/21 08:54

Since no bugs were reported, 2.40.9 is now the official release.
© 2024 lite-C Forums