exec() enhancement: return a handle for tracking

Posted By: AndrewAMD

exec() enhancement: return a handle for tracking - 10/30/18 13:17

jcl,

exec() currently allows me to launch a process synchronously (Mode = 1) or asynchronously (Mode = 0). But if I do it asynchronously, I cannot track the result.

Consider this scenario:

Suppose I want to run a batch of 100 zorro scripts, but I only want to run four or eight at a time (due to my processor having four cores / eight logical threads, and for some reason, the scripts are single-threaded only). In this case, I would ideally like to launch a master zorro script:
* First, it launches four zorros.
* Next, it tracks to see which of the processes finished, maybe once a second.
* Upon one process completion, it launches the next zorro.
* Continue until all 100 scripts are completed.

For this to work, I would need some sort of handle to track each process, such as to detect its status and return code (if applicable).

Can exec() be enhanced to do this?

In the meantime, I suppose I do have two feasible workarounds:
* Run four synchronous batch files simultaneously, each launching zorro over and over again.
* Use the WinAPI directly to create and track process handles.

Thanks,
Andrew
Posted By: Dalla

Re: exec() enhancement: return a handle for tracking - 10/30/18 21:00

Do you really have 100 scripts to run? :-)

Anyway, you can probably do something like this with power shell.

Below is something I created last year, perhaps it can serve as inspiration at least.

Here basically a parallell block will run many commands in parallell (duh!), but it won´t continue with the next parallell block of tasks until every process in the previos parallell block has finished.

Code:
#Add parameter for index (replace=)
#Filter out less trades for OOS, to much garbage currently

Workflow Zorro-Workflow
{


    InlineScript { 
        if([System.IO.File]::Exists("Y:ZorroLogGenerated.csv")) {
            Remove-Item Y:ZorroLogGenerated.csv
        }
        if([System.IO.File]::Exists("Y:ZorroLogGenSortedTest.csv")){
            Remove-Item Y:ZorroLogGenSortedTest.csv
        }
        if([System.IO.File]::Exists("Y:ZorroLogGeneratedFactoryResult.csv")){
            Move-Item -Path Y:ZorroLogGeneratedFactoryResult.csv -Destination Y:ZorroLogGeneratedFactoryResult_$(get-date -f yyyy-MM-dd_HHmm).csv
        }
    }
    
    #Parameters to Zorro are in order: BarPeriod, StopMultiplier, TpMultiplier, LifeTime
    parallel
    {
       InlineScript { 
            Y:ZorroZorro.exe -train -a EUR/USD -d SHORT -i 60 -i 0 -i 0 -i 5 MiningRunner | Out-Null 
            Write-Host "Finished first batch" 
       }
       

       InlineScript { 
            Y:ZorroZorro.exe -train -a EUR/USD -d SHORT -i 60 -i 0 -i 0 -i 10 MiningRunner| Out-Null 
            Write-Host "Finished second batch" 
       }

       InlineScript { 
            Y:ZorroZorro.exe -train -a EUR/USD -d SHORT -i 60 -i 0 -i 0 -i 15 MiningRunner| Out-Null 
            Write-Host "Finished third batch" 
       }
    }
    
    
    parallel
    {    
       InlineScript { 
            Y:ZorroZorro.exe -train -a EUR/USD -d SHORT -i 60 -i 4 -i 2 -i 5 MiningRunner| Out-Null 
            Write-Host "Finished fourth batch" 
       }

       InlineScript { 
            Y:ZorroZorro.exe -train -a EUR/USD -d SHORT -i 60 -i 4 -i 2 -i 10 MiningRunner| Out-Null 
            Write-Host "Finished fifth batch" 
       }

       InlineScript { 
            Y:ZorroZorro.exe -train -a EUR/USD -d SHORT -i 60 -i 4 -i 2 -i 15 MiningRunner| Out-Null 
            Write-Host "Finished sixth batch" 
       }
       

    }       

    parallel
    {    
       InlineScript { 
            Y:ZorroZorro.exe -train -a EUR/USD -d SHORT -i 60 -i 4 -i 4 -i 5 MiningRunner| Out-Null 
            Write-Host "Finished seventh batch" 
       }

       InlineScript { 
            Y:ZorroZorro.exe -train -a EUR/USD -d SHORT -i 60 -i 4 -i 4 -i 10 MiningRunner| Out-Null 
            Write-Host "Finished eighth batch" 
       }

       InlineScript { 
            Y:ZorroZorro.exe -train -a EUR/USD -d SHORT -i 60 -i 4 -i 4 -i 15 MiningRunner| Out-Null 
            Write-Host "Finished ninth batch" 
       }
       

    }  
    

    InlineScript {
        Write-Host "Starting OOS testing"
        Import-Csv Y:ZorroLogGenerated.csv -delimiter "`t" | sort ProfitFactor -Descending  | Export-Csv -Path Y:ZorroLogGenSortedTest.csv -NoTypeInformation
        $csv = Import-Csv Y:ZorroLogGenSortedTest.csv | where {$_.Total -notlike '-*'}

        #Create result file and add headers
        $header = "Asset`tBarPeriod`tStopMultiplier`tTakeProfitMultiplier`tLifeTime`tTrades`tTotal`tWinPerTrade`tProfitFactor`te-Ratio`tStrategy"
        $header | Set-Content 'Y:ZorroLogGeneratedFactoryResult.csv'

        $totalLines = $csv.Count

        #TODO: Add random filename to be able to run OOS tests in parallell
        #probably using #([System.IO.Path]::GetRandomFileName()).Split(�.�)[0]
        foreach ($line in $csv)
        {
            $lineIndex = $csv.IndexOf($line)+1
            $asset = $line.Asset
            $asset = $asset.replace('/','')

            Write-Host "Running OOS for system $lineIndex of $totalLines"
            (Get-Content 'Y:ZorroStrategyGenFactoryTemplate.c') -replace 'replaceMe', $line.Strategy | Set-Content 'Y:ZorroStrategyGenFactoryTest.c'
            Y:ZorroZorro.exe -run -a $line.Asset -d SHORT -i $line.BarPeriod -i $line.StopMultiplier -i $line.TakeProfitMultiplier -i $line.LifeTime GenFactoryTest | Out-Null
            Move-Item -Path Y:ZorroLogGenFactoryTest_$asset.png -Destination Y:ZorroLogGenPlotsGenFactoryTest_$asset_$lineIndex.png
        } 
        Write-Host "OOS testing completed, all done!"
    }
}

Posted By: AndrewAMD

Re: exec() enhancement: return a handle for tracking - 10/30/18 23:45

Originally Posted By: Dalla
Do you really have 100 scripts to run? :-)
Not yet, but I anticipate some similar problems to arise soon. QuantLib can really take a while to emulate options data, and it's not designed to do multi-threading, so I should prefer multi-process, according to Luigi Ballabio.

Thanks for the script. I probably should learn some Powershell... cmd is awkward for this task. Python also looks like a pretty good fit.

I still think I would be writing less scripting code if exec() is upgraded as described.
Posted By: jcl

Re: exec() enhancement: return a handle for tracking - 10/31/18 14:59

In fact exec() already returns the process Id of the created process. This is undocumented, but it should work - try it.
Posted By: AndrewAMD

Re: exec() enhancement: return a handle for tracking - 10/31/18 16:12

Originally Posted By: jcl
In fact exec() already returns the process Id of the created process. This is undocumented, but it should work - try it.


It works! I had to add a little boilerplate code, though.

Code:
#include <default.c>
#include <windows.h>
#define PROCESS_ALL_ACCESS (STANDARD_RIGHTS_REQUIRED | SYNCHRONIZE | 0xFFFF)
#define STATUS_PENDING                   ((DWORD   )0x00000103L)    
#define STILL_ACTIVE                        STATUS_PENDING

int main(){
	DWORD pid = exec("notepad",0,0); // open notepad
	HANDLE h = OpenProcess(PROCESS_ALL_ACCESS,TRUE,pid);
	
	printf("\nhandle: %d",h);
	while(true){
		if(!wait(5000)) break; // wait five seconds
		DWORD code;
		bool good = GetExitCodeProcess(h,&code);
		if(!good){
			int err = GetLastError();
			printf("\nGetExitCodeProcess failed, error %d", GetLastError());
			if (err == 5) printf(": ERROR_ACCESS_DENIED");
			if (err == 6) printf(": ERROR_INVALID_HANDLE");
			break;
		}
		if(code==STILL_ACTIVE){
			printf("\nNotepad is still open...");
		} else {
			printf("\nNotepad is finally closed!");
			break;
		}
	}
	CloseHandle(h);
	return 0;
}

Posted By: AndrewAMD

Re: exec() enhancement: return a handle for tracking - 11/01/18 17:59

Update: I ended up making a C++ implementation after all, it was easy.

It's a basic command-line exe file that loads a batch file of one-liners.

It will either auto-detect your number of threads, or you can manually input number of threads.

Try it out:
1) Unzip the files to a folder somewhere.
2) Try out this command: multibatcher example_batch.bat



Attached picture Capture.PNG
Attached File
multibatcher.zip  (71 downloads)
Posted By: MatPed

Re: exec() enhancement: return a handle for tracking - 11/01/18 18:12

@AndrewAMD In the coding game, you are playing with the PRO! I'm sooooooooooo envious!
Posted By: AndrewAMD

Re: exec() enhancement: return a handle for tracking - 11/01/18 20:36

Running the Trend Test batch file...



Originally Posted By: MatPed
@AndrewAMD In the coding game, you are playing with the PRO! I'm sooooooooooo envious!
Thanks! smile

Attached picture TrendTest.PNG
Posted By: AndrewAMD

Re: exec() enhancement: return a handle for tracking - 05/14/19 16:36

I rewrote the notepad example with some helper functions, for convenience.

Code:
#include <default.c>

// Required for background process (bgp) functions
#include <windows.h>
#define PROCESS_ALL_ACCESS (STANDARD_RIGHTS_REQUIRED | SYNCHRONIZE | 0xFFFF)
#define STATUS_PENDING		((DWORD   )0x00000103L)    
#define STILL_ACTIVE       STATUS_PENDING
#define EXITCODE_ERROR 		200

// Returns bgp handle or 0 if failure.
HANDLE bgp_launch(string Program, string Options);

// Returns STILL_ACTIVE if process is still running. Else closes handle and returns code.
// On error, it returns EXITCODE_ERROR and prints the error code to the log.
DWORD bgp_exit_code(HANDLE h);

int main(){
	HANDLE h = bgp_launch("notepad",0);
	printf("\nhandle: %d",h);
	while(true){
		if(!wait(5000)) break; // wait five seconds
		DWORD code = bgp_exit_code(h);
		if(code==STILL_ACTIVE){
			printf("\nNotepad is still open...");
		} 
		else if (code == EXITCODE_ERROR){
			printf("\nbgp_exit_code encountered an error.");
			break;
		}
		else
		{
			printf("\nNotepad is finally closed!");
			break;
		}
	}
	return 0;
}


HANDLE bgp_launch(string Program, string Options){
	DWORD pid = exec(Program,Options,0);
	if(!pid){
		printf("\nCannot launch: %s %s",Program,ifelse((int)Options,Options,""));
		return 0;
	}
	printf("\nLaunched: %s %s (pid:%d)",Program,ifelse((int)Options,Options,""),pid);
	HANDLE h = OpenProcess(PROCESS_ALL_ACCESS,TRUE,pid);
	if(!h){
		printf("\nCannot get handle for pid:%d",pid);
		return 0;
	}
	return h;
}

bool bgp_exit_code(HANDLE h){
	DWORD code;
	bool good = GetExitCodeProcess(h,&code);
	if(!good){
		int err = GetLastError();
		printf("\nGetExitCodeProcess failed, error %d", GetLastError());
		if (err == 5) printf(": ERROR_ACCESS_DENIED");
		if (err == 6) printf(": ERROR_INVALID_HANDLE");
		return EXITCODE_ERROR;
	}
	if(code!=STILL_ACTIVE){
		CloseHandle(h);
	}
	return code;
}

© 2024 lite-C Forums