Gamestudio Links
Zorro Links
Newest Posts
Specular Shader
by Dooley
Yesterday at 18:17
is 3dgs byDirextX or Opengl?
by Dooley
Yesterday at 18:09
Tensorflow and Python 32 bit
by jcl
Yesterday at 13:28
New Zorro version 2.12
by jcl
05/21/19 17:23
Mixed frequency history (M1+D)
by jcl
05/20/19 12:21
This is my fault-tolerant ATR indicator for you
by OptimusPrime
05/18/19 16:35
missing trade variables
by AndrewAMD
05/18/19 12:37
AUM Magazine
Latest Screens
The Space Between
Pogostuck: Rage With Your Friends
Worst Case Z
AckCon'18 - Lotter vs the World 2 - Preview Release
Who's Online
5 registered (kvm, Michael_Schwarz, Ayumi, Dooley, 1 invisible), 519 Guests and 1 Spider online.
Key: Admin, Global Mod, Mod
Newest Members
Reza65, Pritamchakra, h4h4rd, ccm, Pippo
18224 Registered Users
Topic Options
Rate This Topic
#474688 - 10/30/18 14:17 exec() enhancement: return a handle for tracking *****
AndrewAMD Offline
User

Registered: 02/21/17
Posts: 652
Loc: Chicago
jcl,

exec() currently allows me to launch a process synchronously (Mode = 1) or asynchronously (Mode = 0). But if I do it asynchronously, I cannot track the result.

Consider this scenario:

Suppose I want to run a batch of 100 zorro scripts, but I only want to run four or eight at a time (due to my processor having four cores / eight logical threads, and for some reason, the scripts are single-threaded only). In this case, I would ideally like to launch a master zorro script:
* First, it launches four zorros.
* Next, it tracks to see which of the processes finished, maybe once a second.
* Upon one process completion, it launches the next zorro.
* Continue until all 100 scripts are completed.

For this to work, I would need some sort of handle to track each process, such as to detect its status and return code (if applicable).

Can exec() be enhanced to do this?

In the meantime, I suppose I do have two feasible workarounds:
* Run four synchronous batch files simultaneously, each launching zorro over and over again.
* Use the WinAPI directly to create and track process handles.

Thanks,
Andrew

Top
#474702 - 10/30/18 22:00 Re: exec() enhancement: return a handle for tracking [Re: AndrewAMD]
Dalla Offline
Senior Member

Registered: 02/24/17
Posts: 350
Do you really have 100 scripts to run? :-)

Anyway, you can probably do something like this with power shell.

Below is something I created last year, perhaps it can serve as inspiration at least.

Here basically a parallell block will run many commands in parallell (duh!), but it won´t continue with the next parallell block of tasks until every process in the previos parallell block has finished.

Code:
#Add parameter for index (replace=)
#Filter out less trades for OOS, to much garbage currently

Workflow Zorro-Workflow
{


    InlineScript { 
        if([System.IO.File]::Exists("Y:ZorroLogGenerated.csv")) {
            Remove-Item Y:ZorroLogGenerated.csv
        }
        if([System.IO.File]::Exists("Y:ZorroLogGenSortedTest.csv")){
            Remove-Item Y:ZorroLogGenSortedTest.csv
        }
        if([System.IO.File]::Exists("Y:ZorroLogGeneratedFactoryResult.csv")){
            Move-Item -Path Y:ZorroLogGeneratedFactoryResult.csv -Destination Y:ZorroLogGeneratedFactoryResult_$(get-date -f yyyy-MM-dd_HHmm).csv
        }
    }
    
    #Parameters to Zorro are in order: BarPeriod, StopMultiplier, TpMultiplier, LifeTime
    parallel
    {
       InlineScript { 
            Y:ZorroZorro.exe -train -a EUR/USD -d SHORT -i 60 -i 0 -i 0 -i 5 MiningRunner | Out-Null 
            Write-Host "Finished first batch" 
       }
       

       InlineScript { 
            Y:ZorroZorro.exe -train -a EUR/USD -d SHORT -i 60 -i 0 -i 0 -i 10 MiningRunner| Out-Null 
            Write-Host "Finished second batch" 
       }

       InlineScript { 
            Y:ZorroZorro.exe -train -a EUR/USD -d SHORT -i 60 -i 0 -i 0 -i 15 MiningRunner| Out-Null 
            Write-Host "Finished third batch" 
       }
    }
    
    
    parallel
    {    
       InlineScript { 
            Y:ZorroZorro.exe -train -a EUR/USD -d SHORT -i 60 -i 4 -i 2 -i 5 MiningRunner| Out-Null 
            Write-Host "Finished fourth batch" 
       }

       InlineScript { 
            Y:ZorroZorro.exe -train -a EUR/USD -d SHORT -i 60 -i 4 -i 2 -i 10 MiningRunner| Out-Null 
            Write-Host "Finished fifth batch" 
       }

       InlineScript { 
            Y:ZorroZorro.exe -train -a EUR/USD -d SHORT -i 60 -i 4 -i 2 -i 15 MiningRunner| Out-Null 
            Write-Host "Finished sixth batch" 
       }
       

    }       

    parallel
    {    
       InlineScript { 
            Y:ZorroZorro.exe -train -a EUR/USD -d SHORT -i 60 -i 4 -i 4 -i 5 MiningRunner| Out-Null 
            Write-Host "Finished seventh batch" 
       }

       InlineScript { 
            Y:ZorroZorro.exe -train -a EUR/USD -d SHORT -i 60 -i 4 -i 4 -i 10 MiningRunner| Out-Null 
            Write-Host "Finished eighth batch" 
       }

       InlineScript { 
            Y:ZorroZorro.exe -train -a EUR/USD -d SHORT -i 60 -i 4 -i 4 -i 15 MiningRunner| Out-Null 
            Write-Host "Finished ninth batch" 
       }
       

    }  
    

    InlineScript {
        Write-Host "Starting OOS testing"
        Import-Csv Y:ZorroLogGenerated.csv -delimiter "`t" | sort ProfitFactor -Descending  | Export-Csv -Path Y:ZorroLogGenSortedTest.csv -NoTypeInformation
        $csv = Import-Csv Y:ZorroLogGenSortedTest.csv | where {$_.Total -notlike '-*'}

        #Create result file and add headers
        $header = "Asset`tBarPeriod`tStopMultiplier`tTakeProfitMultiplier`tLifeTime`tTrades`tTotal`tWinPerTrade`tProfitFactor`te-Ratio`tStrategy"
        $header | Set-Content 'Y:ZorroLogGeneratedFactoryResult.csv'

        $totalLines = $csv.Count

        #TODO: Add random filename to be able to run OOS tests in parallell
        #probably using #([System.IO.Path]::GetRandomFileName()).Split(�.�)[0]
        foreach ($line in $csv)
        {
            $lineIndex = $csv.IndexOf($line)+1
            $asset = $line.Asset
            $asset = $asset.replace('/','')

            Write-Host "Running OOS for system $lineIndex of $totalLines"
            (Get-Content 'Y:ZorroStrategyGenFactoryTemplate.c') -replace 'replaceMe', $line.Strategy | Set-Content 'Y:ZorroStrategyGenFactoryTest.c'
            Y:ZorroZorro.exe -run -a $line.Asset -d SHORT -i $line.BarPeriod -i $line.StopMultiplier -i $line.TakeProfitMultiplier -i $line.LifeTime GenFactoryTest | Out-Null
            Move-Item -Path Y:ZorroLogGenFactoryTest_$asset.png -Destination Y:ZorroLogGenPlotsGenFactoryTest_$asset_$lineIndex.png
        } 
        Write-Host "OOS testing completed, all done!"
    }
}


Top
#474703 - 10/31/18 00:45 Re: exec() enhancement: return a handle for tracking [Re: Dalla]
AndrewAMD Offline
User

Registered: 02/21/17
Posts: 652
Loc: Chicago
Originally Posted By: Dalla
Do you really have 100 scripts to run? :-)
Not yet, but I anticipate some similar problems to arise soon. QuantLib can really take a while to emulate options data, and it's not designed to do multi-threading, so I should prefer multi-process, according to Luigi Ballabio.

Thanks for the script. I probably should learn some Powershell... cmd is awkward for this task. Python also looks like a pretty good fit.

I still think I would be writing less scripting code if exec() is upgraded as described.

Top
#474712 - 10/31/18 15:59 Re: exec() enhancement: return a handle for tracking [Re: AndrewAMD]
jcl Online

Chief Engineer

Registered: 07/22/00
Posts: 26869
Loc: Frankfurt
In fact exec() already returns the process Id of the created process. This is undocumented, but it should work - try it.

Top
#474715 - 10/31/18 17:12 Re: exec() enhancement: return a handle for tracking [Re: jcl]
AndrewAMD Offline
User

Registered: 02/21/17
Posts: 652
Loc: Chicago
Originally Posted By: jcl
In fact exec() already returns the process Id of the created process. This is undocumented, but it should work - try it.


It works! I had to add a little boilerplate code, though.

Code:
#include <default.c>
#include <windows.h>
#define PROCESS_ALL_ACCESS (STANDARD_RIGHTS_REQUIRED | SYNCHRONIZE | 0xFFFF)
#define STATUS_PENDING                   ((DWORD   )0x00000103L)    
#define STILL_ACTIVE                        STATUS_PENDING

int main(){
	DWORD pid = exec("notepad",0,0); // open notepad
	HANDLE h = OpenProcess(PROCESS_ALL_ACCESS,TRUE,pid);
	
	printf("\nhandle: %d",h);
	while(true){
		if(!wait(5000)) break; // wait five seconds
		DWORD code;
		bool good = GetExitCodeProcess(h,&code);
		if(!good){
			int err = GetLastError();
			printf("\nGetExitCodeProcess failed, error %d", GetLastError());
			if (err == 5) printf(": ERROR_ACCESS_DENIED");
			if (err == 6) printf(": ERROR_INVALID_HANDLE");
			break;
		}
		if(code==STILL_ACTIVE){
			printf("\nNotepad is still open...");
		} else {
			printf("\nNotepad is finally closed!");
			break;
		}
	}
	CloseHandle(h);
	return 0;
}


Top
#474730 - 11/01/18 18:59 Re: exec() enhancement: return a handle for tracking [Re: AndrewAMD]
AndrewAMD Offline
User

Registered: 02/21/17
Posts: 652
Loc: Chicago
Update: I ended up making a C++ implementation after all, it was easy.

It's a basic command-line exe file that loads a batch file of one-liners.

It will either auto-detect your number of threads, or you can manually input number of threads.

Try it out:
1) Unzip the files to a folder somewhere.
2) Try out this command: multibatcher example_batch.bat



Attachments
Capture.PNG (216 downloads)
multibatcher.zip (22 downloads)


Top
#474731 - 11/01/18 19:12 Re: exec() enhancement: return a handle for tracking [Re: AndrewAMD]
MatPed Offline
User

Registered: 02/28/15
Posts: 554
Loc: Milano, Italy
@AndrewAMD In the coding game, you are playing with the PRO! I'm sooooooooooo envious!


Edited by MatPed (11/01/18 19:14)

Top
#474734 - 11/01/18 21:36 Re: exec() enhancement: return a handle for tracking [Re: MatPed]
AndrewAMD Offline
User

Registered: 02/21/17
Posts: 652
Loc: Chicago
Running the Trend Test batch file...



Originally Posted By: MatPed
@AndrewAMD In the coding game, you are playing with the PRO! I'm sooooooooooo envious!
Thanks! smile


Attachments
TrendTest.PNG (226 downloads)


Top
#477065 - 05/14/19 18:36 Re: exec() enhancement: return a handle for tracking [Re: AndrewAMD]
AndrewAMD Offline
User

Registered: 02/21/17
Posts: 652
Loc: Chicago
I rewrote the notepad example with some helper functions, for convenience.

Code:
#include <default.c>

// Required for background process (bgp) functions
#include <windows.h>
#define PROCESS_ALL_ACCESS (STANDARD_RIGHTS_REQUIRED | SYNCHRONIZE | 0xFFFF)
#define STATUS_PENDING		((DWORD   )0x00000103L)    
#define STILL_ACTIVE       STATUS_PENDING
#define EXITCODE_ERROR 		200

// Returns bgp handle or 0 if failure.
HANDLE bgp_launch(string Program, string Options);

// Returns STILL_ACTIVE if process is still running. Else closes handle and returns code.
// On error, it returns EXITCODE_ERROR and prints the error code to the log.
DWORD bgp_exit_code(HANDLE h);

int main(){
	HANDLE h = bgp_launch("notepad",0);
	printf("\nhandle: %d",h);
	while(true){
		if(!wait(5000)) break; // wait five seconds
		DWORD code = bgp_exit_code(h);
		if(code==STILL_ACTIVE){
			printf("\nNotepad is still open...");
		} 
		else if (code == EXITCODE_ERROR){
			printf("\nbgp_exit_code encountered an error.");
			break;
		}
		else
		{
			printf("\nNotepad is finally closed!");
			break;
		}
	}
	return 0;
}


HANDLE bgp_launch(string Program, string Options){
	DWORD pid = exec(Program,Options,0);
	if(!pid){
		printf("\nCannot launch: %s %s",Program,ifelse((int)Options,Options,""));
		return 0;
	}
	printf("\nLaunched: %s %s (pid:%d)",Program,ifelse((int)Options,Options,""),pid);
	HANDLE h = OpenProcess(PROCESS_ALL_ACCESS,TRUE,pid);
	if(!h){
		printf("\nCannot get handle for pid:%d",pid);
		return 0;
	}
	return h;
}

bool bgp_exit_code(HANDLE h){
	DWORD code;
	bool good = GetExitCodeProcess(h,&code);
	if(!good){
		int err = GetLastError();
		printf("\nGetExitCodeProcess failed, error %d", GetLastError());
		if (err == 5) printf(": ERROR_ACCESS_DENIED");
		if (err == 6) printf(": ERROR_INVALID_HANDLE");
		return EXITCODE_ERROR;
	}
	if(code!=STILL_ACTIVE){
		CloseHandle(h);
	}
	return code;
}


Top



Gamestudio download | chip programmers | Zorro platform | shop | Data Protection Policy

oP group Germany GmbH | Birkenstr. 25-27 | 63549 Ronneburg / Germany | info (at) opgroup.de