Gamestudio Links
Zorro Links
Newest Posts
Blobsculptor tools and objects download here
by NeoDumont. 03/28/24 03:01
Issue with Multi-Core WFO Training
by aliswee. 03/24/24 20:20
Why Zorro supports up to 72 cores?
by Edgar_Herrera. 03/23/24 21:41
Zorro Trader GPT
by TipmyPip. 03/06/24 09:27
VSCode instead of SED
by 3run. 03/01/24 19:06
AUM Magazine
Latest Screens
The Bible Game
A psychological thriller game
SHADOW (2014)
DEAD TASTE
Who's Online Now
3 registered members (AndrewAMD, Quad, TipmyPip), 865 guests, and 6 spiders.
Key: Admin, Global Mod, Mod
Newest Members
sakolin, rajesh7827, juergen_wue, NITRO_FOREVER, jack0roses
19043 Registered Users
Previous Thread
Next Thread
Print Thread
Rating: 5
exec() enhancement: return a handle for tracking #474688
10/30/18 13:17
10/30/18 13:17
Joined: Feb 2017
Posts: 1,718
Chicago
AndrewAMD Online OP
Serious User
AndrewAMD  Online OP
Serious User

Joined: Feb 2017
Posts: 1,718
Chicago
jcl,

exec() currently allows me to launch a process synchronously (Mode = 1) or asynchronously (Mode = 0). But if I do it asynchronously, I cannot track the result.

Consider this scenario:

Suppose I want to run a batch of 100 zorro scripts, but I only want to run four or eight at a time (due to my processor having four cores / eight logical threads, and for some reason, the scripts are single-threaded only). In this case, I would ideally like to launch a master zorro script:
* First, it launches four zorros.
* Next, it tracks to see which of the processes finished, maybe once a second.
* Upon one process completion, it launches the next zorro.
* Continue until all 100 scripts are completed.

For this to work, I would need some sort of handle to track each process, such as to detect its status and return code (if applicable).

Can exec() be enhanced to do this?

In the meantime, I suppose I do have two feasible workarounds:
* Run four synchronous batch files simultaneously, each launching zorro over and over again.
* Use the WinAPI directly to create and track process handles.

Thanks,
Andrew

Re: exec() enhancement: return a handle for tracking [Re: AndrewAMD] #474702
10/30/18 21:00
10/30/18 21:00
Joined: Feb 2017
Posts: 369
D
Dalla Offline
Senior Member
Dalla  Offline
Senior Member
D

Joined: Feb 2017
Posts: 369
Do you really have 100 scripts to run? :-)

Anyway, you can probably do something like this with power shell.

Below is something I created last year, perhaps it can serve as inspiration at least.

Here basically a parallell block will run many commands in parallell (duh!), but it won´t continue with the next parallell block of tasks until every process in the previos parallell block has finished.

Code:
#Add parameter for index (replace=)
#Filter out less trades for OOS, to much garbage currently

Workflow Zorro-Workflow
{


    InlineScript { 
        if([System.IO.File]::Exists("Y:ZorroLogGenerated.csv")) {
            Remove-Item Y:ZorroLogGenerated.csv
        }
        if([System.IO.File]::Exists("Y:ZorroLogGenSortedTest.csv")){
            Remove-Item Y:ZorroLogGenSortedTest.csv
        }
        if([System.IO.File]::Exists("Y:ZorroLogGeneratedFactoryResult.csv")){
            Move-Item -Path Y:ZorroLogGeneratedFactoryResult.csv -Destination Y:ZorroLogGeneratedFactoryResult_$(get-date -f yyyy-MM-dd_HHmm).csv
        }
    }
    
    #Parameters to Zorro are in order: BarPeriod, StopMultiplier, TpMultiplier, LifeTime
    parallel
    {
       InlineScript { 
            Y:ZorroZorro.exe -train -a EUR/USD -d SHORT -i 60 -i 0 -i 0 -i 5 MiningRunner | Out-Null 
            Write-Host "Finished first batch" 
       }
       

       InlineScript { 
            Y:ZorroZorro.exe -train -a EUR/USD -d SHORT -i 60 -i 0 -i 0 -i 10 MiningRunner| Out-Null 
            Write-Host "Finished second batch" 
       }

       InlineScript { 
            Y:ZorroZorro.exe -train -a EUR/USD -d SHORT -i 60 -i 0 -i 0 -i 15 MiningRunner| Out-Null 
            Write-Host "Finished third batch" 
       }
    }
    
    
    parallel
    {    
       InlineScript { 
            Y:ZorroZorro.exe -train -a EUR/USD -d SHORT -i 60 -i 4 -i 2 -i 5 MiningRunner| Out-Null 
            Write-Host "Finished fourth batch" 
       }

       InlineScript { 
            Y:ZorroZorro.exe -train -a EUR/USD -d SHORT -i 60 -i 4 -i 2 -i 10 MiningRunner| Out-Null 
            Write-Host "Finished fifth batch" 
       }

       InlineScript { 
            Y:ZorroZorro.exe -train -a EUR/USD -d SHORT -i 60 -i 4 -i 2 -i 15 MiningRunner| Out-Null 
            Write-Host "Finished sixth batch" 
       }
       

    }       

    parallel
    {    
       InlineScript { 
            Y:ZorroZorro.exe -train -a EUR/USD -d SHORT -i 60 -i 4 -i 4 -i 5 MiningRunner| Out-Null 
            Write-Host "Finished seventh batch" 
       }

       InlineScript { 
            Y:ZorroZorro.exe -train -a EUR/USD -d SHORT -i 60 -i 4 -i 4 -i 10 MiningRunner| Out-Null 
            Write-Host "Finished eighth batch" 
       }

       InlineScript { 
            Y:ZorroZorro.exe -train -a EUR/USD -d SHORT -i 60 -i 4 -i 4 -i 15 MiningRunner| Out-Null 
            Write-Host "Finished ninth batch" 
       }
       

    }  
    

    InlineScript {
        Write-Host "Starting OOS testing"
        Import-Csv Y:ZorroLogGenerated.csv -delimiter "`t" | sort ProfitFactor -Descending  | Export-Csv -Path Y:ZorroLogGenSortedTest.csv -NoTypeInformation
        $csv = Import-Csv Y:ZorroLogGenSortedTest.csv | where {$_.Total -notlike '-*'}

        #Create result file and add headers
        $header = "Asset`tBarPeriod`tStopMultiplier`tTakeProfitMultiplier`tLifeTime`tTrades`tTotal`tWinPerTrade`tProfitFactor`te-Ratio`tStrategy"
        $header | Set-Content 'Y:ZorroLogGeneratedFactoryResult.csv'

        $totalLines = $csv.Count

        #TODO: Add random filename to be able to run OOS tests in parallell
        #probably using #([System.IO.Path]::GetRandomFileName()).Split(�.�)[0]
        foreach ($line in $csv)
        {
            $lineIndex = $csv.IndexOf($line)+1
            $asset = $line.Asset
            $asset = $asset.replace('/','')

            Write-Host "Running OOS for system $lineIndex of $totalLines"
            (Get-Content 'Y:ZorroStrategyGenFactoryTemplate.c') -replace 'replaceMe', $line.Strategy | Set-Content 'Y:ZorroStrategyGenFactoryTest.c'
            Y:ZorroZorro.exe -run -a $line.Asset -d SHORT -i $line.BarPeriod -i $line.StopMultiplier -i $line.TakeProfitMultiplier -i $line.LifeTime GenFactoryTest | Out-Null
            Move-Item -Path Y:ZorroLogGenFactoryTest_$asset.png -Destination Y:ZorroLogGenPlotsGenFactoryTest_$asset_$lineIndex.png
        } 
        Write-Host "OOS testing completed, all done!"
    }
}


Re: exec() enhancement: return a handle for tracking [Re: Dalla] #474703
10/30/18 23:45
10/30/18 23:45
Joined: Feb 2017
Posts: 1,718
Chicago
AndrewAMD Online OP
Serious User
AndrewAMD  Online OP
Serious User

Joined: Feb 2017
Posts: 1,718
Chicago
Originally Posted By: Dalla
Do you really have 100 scripts to run? :-)
Not yet, but I anticipate some similar problems to arise soon. QuantLib can really take a while to emulate options data, and it's not designed to do multi-threading, so I should prefer multi-process, according to Luigi Ballabio.

Thanks for the script. I probably should learn some Powershell... cmd is awkward for this task. Python also looks like a pretty good fit.

I still think I would be writing less scripting code if exec() is upgraded as described.

Re: exec() enhancement: return a handle for tracking [Re: AndrewAMD] #474712
10/31/18 14:59
10/31/18 14:59
Joined: Jul 2000
Posts: 27,977
Frankfurt
jcl Offline

Chief Engineer
jcl  Offline

Chief Engineer

Joined: Jul 2000
Posts: 27,977
Frankfurt
In fact exec() already returns the process Id of the created process. This is undocumented, but it should work - try it.

Re: exec() enhancement: return a handle for tracking [Re: jcl] #474715
10/31/18 16:12
10/31/18 16:12
Joined: Feb 2017
Posts: 1,718
Chicago
AndrewAMD Online OP
Serious User
AndrewAMD  Online OP
Serious User

Joined: Feb 2017
Posts: 1,718
Chicago
Originally Posted By: jcl
In fact exec() already returns the process Id of the created process. This is undocumented, but it should work - try it.


It works! I had to add a little boilerplate code, though.

Code:
#include <default.c>
#include <windows.h>
#define PROCESS_ALL_ACCESS (STANDARD_RIGHTS_REQUIRED | SYNCHRONIZE | 0xFFFF)
#define STATUS_PENDING                   ((DWORD   )0x00000103L)    
#define STILL_ACTIVE                        STATUS_PENDING

int main(){
	DWORD pid = exec("notepad",0,0); // open notepad
	HANDLE h = OpenProcess(PROCESS_ALL_ACCESS,TRUE,pid);
	
	printf("\nhandle: %d",h);
	while(true){
		if(!wait(5000)) break; // wait five seconds
		DWORD code;
		bool good = GetExitCodeProcess(h,&code);
		if(!good){
			int err = GetLastError();
			printf("\nGetExitCodeProcess failed, error %d", GetLastError());
			if (err == 5) printf(": ERROR_ACCESS_DENIED");
			if (err == 6) printf(": ERROR_INVALID_HANDLE");
			break;
		}
		if(code==STILL_ACTIVE){
			printf("\nNotepad is still open...");
		} else {
			printf("\nNotepad is finally closed!");
			break;
		}
	}
	CloseHandle(h);
	return 0;
}


Re: exec() enhancement: return a handle for tracking [Re: AndrewAMD] #474730
11/01/18 17:59
11/01/18 17:59
Joined: Feb 2017
Posts: 1,718
Chicago
AndrewAMD Online OP
Serious User
AndrewAMD  Online OP
Serious User

Joined: Feb 2017
Posts: 1,718
Chicago
Update: I ended up making a C++ implementation after all, it was easy.

It's a basic command-line exe file that loads a batch file of one-liners.

It will either auto-detect your number of threads, or you can manually input number of threads.

Try it out:
1) Unzip the files to a folder somewhere.
2) Try out this command: multibatcher example_batch.bat


Attached Files Capture.PNG
multibatcher.zip (70 downloads)
Re: exec() enhancement: return a handle for tracking [Re: AndrewAMD] #474731
11/01/18 18:12
11/01/18 18:12
Joined: Feb 2015
Posts: 652
Milano, Italy
M
MatPed Offline
User
MatPed  Offline
User
M

Joined: Feb 2015
Posts: 652
Milano, Italy
@AndrewAMD In the coding game, you are playing with the PRO! I'm sooooooooooo envious!

Last edited by MatPed; 11/01/18 18:14.
Re: exec() enhancement: return a handle for tracking [Re: MatPed] #474734
11/01/18 20:36
11/01/18 20:36
Joined: Feb 2017
Posts: 1,718
Chicago
AndrewAMD Online OP
Serious User
AndrewAMD  Online OP
Serious User

Joined: Feb 2017
Posts: 1,718
Chicago
Running the Trend Test batch file...



Originally Posted By: MatPed
@AndrewAMD In the coding game, you are playing with the PRO! I'm sooooooooooo envious!
Thanks! smile

Attached Files TrendTest.PNG
Re: exec() enhancement: return a handle for tracking [Re: AndrewAMD] #477065
05/14/19 16:36
05/14/19 16:36
Joined: Feb 2017
Posts: 1,718
Chicago
AndrewAMD Online OP
Serious User
AndrewAMD  Online OP
Serious User

Joined: Feb 2017
Posts: 1,718
Chicago
I rewrote the notepad example with some helper functions, for convenience.

Code:
#include <default.c>

// Required for background process (bgp) functions
#include <windows.h>
#define PROCESS_ALL_ACCESS (STANDARD_RIGHTS_REQUIRED | SYNCHRONIZE | 0xFFFF)
#define STATUS_PENDING		((DWORD   )0x00000103L)    
#define STILL_ACTIVE       STATUS_PENDING
#define EXITCODE_ERROR 		200

// Returns bgp handle or 0 if failure.
HANDLE bgp_launch(string Program, string Options);

// Returns STILL_ACTIVE if process is still running. Else closes handle and returns code.
// On error, it returns EXITCODE_ERROR and prints the error code to the log.
DWORD bgp_exit_code(HANDLE h);

int main(){
	HANDLE h = bgp_launch("notepad",0);
	printf("\nhandle: %d",h);
	while(true){
		if(!wait(5000)) break; // wait five seconds
		DWORD code = bgp_exit_code(h);
		if(code==STILL_ACTIVE){
			printf("\nNotepad is still open...");
		} 
		else if (code == EXITCODE_ERROR){
			printf("\nbgp_exit_code encountered an error.");
			break;
		}
		else
		{
			printf("\nNotepad is finally closed!");
			break;
		}
	}
	return 0;
}


HANDLE bgp_launch(string Program, string Options){
	DWORD pid = exec(Program,Options,0);
	if(!pid){
		printf("\nCannot launch: %s %s",Program,ifelse((int)Options,Options,""));
		return 0;
	}
	printf("\nLaunched: %s %s (pid:%d)",Program,ifelse((int)Options,Options,""),pid);
	HANDLE h = OpenProcess(PROCESS_ALL_ACCESS,TRUE,pid);
	if(!h){
		printf("\nCannot get handle for pid:%d",pid);
		return 0;
	}
	return h;
}

bool bgp_exit_code(HANDLE h){
	DWORD code;
	bool good = GetExitCodeProcess(h,&code);
	if(!good){
		int err = GetLastError();
		printf("\nGetExitCodeProcess failed, error %d", GetLastError());
		if (err == 5) printf(": ERROR_ACCESS_DENIED");
		if (err == 6) printf(": ERROR_INVALID_HANDLE");
		return EXITCODE_ERROR;
	}
	if(code!=STILL_ACTIVE){
		CloseHandle(h);
	}
	return code;
}



Moderated by  Petra 

Gamestudio download | chip programmers | Zorro platform | shop | Data Protection Policy

oP group Germany GmbH | Birkenstr. 25-27 | 63549 Ronneburg / Germany | info (at) opgroup.de

Powered by UBB.threads™ PHP Forum Software 7.7.1