Gamestudio Links
Zorro Links
Newest Posts
Deeplearning Script
by wolfi. 02/26/24 12:46
Sam Foster Sound | Experienced Game Composer for Hire
by titanicpiano14. 02/22/24 16:22
AssetAdd() vs. modern asset list?
by jcl. 02/21/24 15:01
How many still using A8
by Aku_Aku. 02/20/24 12:18
Zorro Trader GPT
by TipmyPip. 02/16/24 06:53
Support PEEK for time and calendar functions
by AndrewAMD. 02/14/24 22:23
AUM Magazine
Latest Screens
The Bible Game
A psychological thriller game
SHADOW (2014)
DEAD TASTE
Who's Online Now
2 registered members (VoroneTZ, PeroPero), 667 guests, and 7 spiders.
Key: Admin, Global Mod, Mod
Newest Members
sakolin, rajesh7827, juergen_wue, NITRO_FOREVER, jack0roses
19043 Registered Users
Previous Thread
Next Thread
Print Thread
Rate Thread
neural function for Python to [Train] #487842
10/11/23 19:14
10/11/23 19:14
Joined: Aug 2023
Posts: 19
I
izorro Offline OP
Newbie
izorro  Offline OP
Newbie
I

Joined: Aug 2023
Posts: 19
In the python-bridge documentation it is mentioned a neural function will be provided in a future zorro release. Would be nice to have it soon.
In the mean time I am trying to set it up (most likely incorrectly), and would appreciate some help. Zorro manual, FH/RW blogs - they all lean towards R frown


This is what im coming up so far with. I have not properly tested yet. I just want someone who had been this route to confirm i am going in the correct direction.

The python script.

Code
import numpy as np
from tensorflow import keras
from tensorflow.keras.layers import Dense, Dropout
from tensorflow.keras.optimizers import RMSprop

models = []

def neural_train(model, XY):
    X = np.array(XY.iloc[:, :-1])
    Y = XY.iloc[:, -1].apply(lambda x: 1 if x > 0 else 0)

    model = keras.models.Sequential()
    model.add(Dense(30, activation='relu', input_shape=(X.shape[1],)))
    model.add(Dropout(0.2))
    model.add(Dense(1, activation='sigmoid'))

    model.compile(loss='binary_crossentropy',
                  optimizer=RMSprop(),
                  metrics=['accuracy'])

    model.fit(X, Y, epochs=20, batch_size=20, validation_split=0, shuffle=False)

    models.append(model)

def neural_predict(model, X):
    if isinstance(X, list):
        X = np.array(X)
    elif isinstance(X, np.ndarray) and X.ndim == 1:
        X = X.reshape(1, -1)

    Y = models[model].predict(X)
    return [1 if y > 0.5 else 0 for y in Y]

def neural_save(name):
    serialized_models = [model.to_json() for model in models]
    with open(name, 'w') as file:
        for serialized_model in serialized_models:
            file.write(serialized_model + '\n')

def neural_load(name):
    global models
    models = []
    with open(name, 'r') as file:
        for line in file:
            model = keras.models.model_from_json(line.strip())
            models.append(model)

def neural_init():
    np.random.seed(365)
    global models
    models = []

# Example usage:
# neural_init()
# neural_train(0, XY)
# result = neural_predict(0, X)
# neural_save('models.txt')
# neural_load('models.txt')



The `neural` function

Code
var neural(int Status, int model, int NumSignals, void* Data)
{
  if(!wait(0)) return 0;
  // open a Python script with the same name as the strategy script  
  if(Status == NEURAL_INIT) {
    if(!pyStart(strf("%s.py",Script),1)) return 0;
    pyX("neural_init()");
    return 1;
  }
  // export batch training samples and call the Python training function
  if(Status == NEURAL_TRAIN) {
    string name = strf("Data\\signals%i.csv",Core);
    file_write(name,Data,0);
    pyX(strf("XY = pandas.read_csv('%s%s',header = None)",slash(ZorroFolder),slash(name)));
    pySet("AlgoVar",AlgoVar,8);
    if(!pyX(strf("neural_train(%i,XY)",model+1)))
      return 0;
    return 1;
  }
  // predict the target with the Python predict function
  if(Status == NEURAL_PREDICT) {
    pySet("AlgoVar",AlgoVar,8);
    pySet("X",(double*)Data,NumSignals);
    pyX(strf("Y = neural_predict(%i,X)",model+1));
    return pyVar("Y[0]");
  }
  // save all trained models  
  if(Status == NEURAL_SAVE) {
    print(TO_ANY,"\nStore %s",strrchr(Data,'\\')+1);
    return pyX(strf("neural_save('%s')",slash(Data)));
  }
  // load all trained models  
  if(Status == NEURAL_LOAD) {
    printf("\nLoad %s",strrchr(Data,'\\')+1);
    return pyX(strf("neural_load('%s')",slash(Data)));
  }
  return 1;
}

Re: neural function for Python to [Train] [Re: izorro] #487844
10/12/23 16:32
10/12/23 16:32
Joined: Aug 2023
Posts: 19
I
izorro Offline OP
Newbie
izorro  Offline OP
Newbie
I

Joined: Aug 2023
Posts: 19
One mistake is that I have to run pystart with mode 1/2 so it works with libraries that uses numpy.

Re: neural function for Python to [Train] [Re: izorro] #487939
11/26/23 09:37
11/26/23 09:37
Joined: Sep 2017
Posts: 72
T
TipmyPip Offline
Junior Member
TipmyPip  Offline
Junior Member
T

Joined: Sep 2017
Posts: 72
1. Data Synchronization and Formatting:
Ensure the data exported from Zorro for training matches the format expected by the Python script.
For predictions, format the input data in lite-C to align with the neural network's input structure in Python.

2. Efficient Python Integration:
Optimize the communication between lite-C and Python. Consider using JSON or similar formats for complex data structures.
Evaluate the use of sockets or REST APIs for faster, more reliable data exchange, especially in real-time trading scenarios.

3. Model Complexity and Customization:
Enhance the Python neural network model with additional layers, different architectures (e.g., LSTM for time series), or more sophisticated features.
Experiment with different optimization algorithms and loss functions in the neural network to improve performance.

4. Robust Error Handling:
Implement comprehensive error handling in both lite-C and Python scripts. Ensure that the system can gracefully handle and recover from unexpected scenarios.

5. Performance Optimization:
Monitor and optimize the performance of both the lite-C and Python scripts. Consider the computational complexity and response time, especially for live trading.

6. Scalability and Modular Design:
Design the system to be scalable. Modularize the code so that new features, assets, or trading strategies can be added easily.

7. Advanced Data Analysis Techniques:
Implement more advanced data analysis and preprocessing techniques in Python before training the model, such as feature scaling, normalization, or using PCA for dimensionality reduction.

8. Dynamic Model Training and Updating:
Automate the process of retraining the model with new data regularly to keep the model up-to-date with market changes.

9. Testing and Validation:
Conduct thorough backtesting in Zorro using historical data. Validate the model's effectiveness under different market conditions.

10. Comprehensive Documentation:
Maintain detailed documentation covering all aspects of the system, including data flow, model specifications, integration details, and usage instructions.

11. User Interaction and Feedback:
Incorporate user feedback mechanisms in Zorro to adjust parameters or strategies dynamically. This could include sliders or input fields for real-time parameter tuning.

12. Legal and Compliance Considerations:
Stay informed about regulatory requirements related to automated trading in different markets to ensure compliance.

By focusing on these areas, you can develop more sophisticated, efficient, and effective trading solutions that intrigue and benefit users of the Zorro Trader lite-C Coding Mentor. (link provided below)


ZorroTraderGPT - https://bit.ly/3Gbsm4S
Re: neural function for Python to [Train] [Re: TipmyPip] #487950
11/30/23 12:53
11/30/23 12:53
Joined: Dec 2019
Posts: 53
ozgur Offline
Junior Member
ozgur  Offline
Junior Member

Joined: Dec 2019
Posts: 53
Originally Posted by TipmyPip
1. Data Synchronization and Formatting:
Ensure the data exported from Zorro for training matches the format expected by the Python script.
For predictions, format the input data in lite-C to align with the neural network's input structure in Python.

2. Efficient Python Integration:
Optimize the communication between lite-C and Python. Consider using JSON or similar formats for complex data structures.
Evaluate the use of sockets or REST APIs for faster, more reliable data exchange, especially in real-time trading scenarios.

3. Model Complexity and Customization:
Enhance the Python neural network model with additional layers, different architectures (e.g., LSTM for time series), or more sophisticated features.
Experiment with different optimization algorithms and loss functions in the neural network to improve performance.

4. Robust Error Handling:
Implement comprehensive error handling in both lite-C and Python scripts. Ensure that the system can gracefully handle and recover from unexpected scenarios.

5. Performance Optimization:
Monitor and optimize the performance of both the lite-C and Python scripts. Consider the computational complexity and response time, especially for live trading.

6. Scalability and Modular Design:
Design the system to be scalable. Modularize the code so that new features, assets, or trading strategies can be added easily.

7. Advanced Data Analysis Techniques:
Implement more advanced data analysis and preprocessing techniques in Python before training the model, such as feature scaling, normalization, or using PCA for dimensionality reduction.

8. Dynamic Model Training and Updating:
Automate the process of retraining the model with new data regularly to keep the model up-to-date with market changes.

9. Testing and Validation:
Conduct thorough backtesting in Zorro using historical data. Validate the model's effectiveness under different market conditions.

10. Comprehensive Documentation:
Maintain detailed documentation covering all aspects of the system, including data flow, model specifications, integration details, and usage instructions.

11. User Interaction and Feedback:
Incorporate user feedback mechanisms in Zorro to adjust parameters or strategies dynamically. This could include sliders or input fields for real-time parameter tuning.

12. Legal and Compliance Considerations:
Stay informed about regulatory requirements related to automated trading in different markets to ensure compliance.

By focusing on these areas, you can develop more sophisticated, efficient, and effective trading solutions that intrigue and benefit users of the Zorro Trader lite-C Coding Mentor. (link provided below)


What is the point of posting generic GPT responses? You are creating information pollution.

Maybe you are a bot.

Last edited by ozgur; 11/30/23 12:54.
Re: neural function for Python to [Train] [Re: ozgur] #487951
11/30/23 15:45
11/30/23 15:45
Joined: Sep 2017
Posts: 72
T
TipmyPip Offline
Junior Member
TipmyPip  Offline
Junior Member
T

Joined: Sep 2017
Posts: 72
Have you tried the one of the solutions?

Can you present any one of them, please?

Do you know what is the relation of suggestions to the above source code related to the Neural Function?


ZorroTraderGPT - https://bit.ly/3Gbsm4S
Re: neural function for Python to [Train] [Re: izorro] #487952
11/30/23 20:24
11/30/23 20:24
Joined: Dec 2019
Posts: 53
ozgur Offline
Junior Member
ozgur  Offline
Junior Member

Joined: Dec 2019
Posts: 53
Yeah, you are a bot. A stupid one in fact. Reported.

Re: neural function for Python to [Train] [Re: ozgur] #487954
12/01/23 05:26
12/01/23 05:26
Joined: Sep 2017
Posts: 72
T
TipmyPip Offline
Junior Member
TipmyPip  Offline
Junior Member
T

Joined: Sep 2017
Posts: 72
Thank you for reporting me, if I am so stupid in the measure of your clever mind, not answering non of my questions will most likely, you are not talking in relevance to my questions, instead of helping here, you seem to think by accusing others of what you are not capable of, makes you wrong.

And please inform me, if you please, Dear Precious Zorro Tradre User, what would it take to make you kinder and generous to other users?

Last edited by TipmyPip; 12/01/23 22:06.

ZorroTraderGPT - https://bit.ly/3Gbsm4S
Re: neural function for Python to [Train] [Re: izorro] #487956
12/01/23 14:47
12/01/23 14:47
Joined: Sep 2017
Posts: 72
T
TipmyPip Offline
Junior Member
TipmyPip  Offline
Junior Member
T

Joined: Sep 2017
Posts: 72
How about :

Code
// Optimization parameters - set these as default values
var learning_rate = 0.001;
var num_neurons = 50;
var dropout_rate = 0.3;
var epochs = 100;
var batch_size = 32;

// Function to handle neural network operations
var neural(int Status, int model, int NumSignals, void* Data)
{
    if(!wait(0)) return 0;

    // Optimization parameters - adjust these as needed within optimize calls
    learning_rate = optimize(0.001, 0.0001, 0.01, 0.0001);
    num_neurons = optimize(50, 10, 100, 10);
    dropout_rate = optimize(0.3, 0.1, 0.5, 0.1);
    epochs = optimize(100, 50, 200, 50);
    batch_size = optimize(32, 16, 64, 16);

    // open a Python script with the same name as the strategy script  
    if(Status == NEURAL_INIT) {
        if(!pyStart(strf("%s.py",Script),1)) return 0;
        pyX("neural_init()");
        return 1;
    }

    // export batch training samples and call the Python training function
    if(Status == NEURAL_TRAIN) {
        string name = strf("Data\\signals%i.csv",Core);
        file_write(name,Data,0);
        pyX(strf("XY = pandas.read_csv('%s%s',header = None)",slash(ZorroFolder),slash(name)));
        pySet("AlgoVar",AlgoVar,8);

        // Pass optimization parameters to Python
        pySet("learning_rate", &learning_rate, 1);
        pySet("num_neurons", &num_neurons, 1);
        pySet("dropout_rate", &dropout_rate, 1);
        pySet("epochs", &epochs, 1);
        pySet("batch_size", &batch_size, 1);

        if(!pyX(strf("neural_train(%i,XY,learning_rate,num_neurons,dropout_rate,epochs,batch_size)", model+1)))
            return 0;
        return 1;
    }

    // predict the target with the Python predict function
    if(Status == NEURAL_PREDICT) {
        pySet("AlgoVar",AlgoVar,8);
        pySet("X",(double*)Data,NumSignals);
        pyX(strf("Y = neural_predict(%i,X)", model+1));
        return pyVar("Y[0]");
    }

    // save all trained models  
    if(Status == NEURAL_SAVE) {
        print(TO_ANY,"\nStore %s", strrchr(Data,'\\')+1);
        return pyX(strf("neural_save('%s')", slash(Data)));
    }

    // load all trained models  
    if(Status == NEURAL_LOAD) {
        printf("\nLoad %s", strrchr(Data,'\\')+1);
        return pyX(strf("neural_load('%s')", slash(Data)));
    }

    return 1;
}



And a new python code:

Code
import numpy as np
from tensorflow import keras
from tensorflow.keras.layers import Dense, Dropout
from tensorflow.keras.optimizers import RMSprop

models = []

def neural_train(model, XY, learning_rate, num_neurons, dropout_rate, epochs, batch_size):
    X = np.array(XY.iloc[:, :-1])
    Y = XY.iloc[:, -1].apply(lambda x: 1 if x > 0 else 0)

    model = keras.models.Sequential()
    model.add(Dense(num_neurons, activation='relu', input_shape=(X.shape[1],)))
    model.add(Dropout(dropout_rate))
    model.add(Dense(1, activation='sigmoid'))

    model.compile(loss='binary_crossentropy',
                  optimizer=RMSprop(learning_rate=learning_rate),
                  metrics=['accuracy'])

    model.fit(X, Y, epochs=epochs, batch_size=batch_size, validation_split=0, shuffle=False)

    models.append(model)

def neural_predict(model, X):
    if isinstance(X, list):
        X = np.array(X)
    elif isinstance(X, np.ndarray) and X.ndim == 1:
        X = X.reshape(1, -1)

    Y = models[model].predict(X)
    return [1 if y > 0.5 else 0 for y in Y]

def neural_save(name):
    serialized_models = [model.to_json() for model in models]
    with open(name, 'w') as file:
        for serialized_model in serialized_models:
            file.write(serialized_model + '\n')

def neural_load(name):
    global models
    models = []
    with open(name, 'r') as file:
        for line in file:
            model = keras.models.model_from_json(line.strip())
            models.append(model)

def neural_init():
    np.random.seed(365)
    global models
    models = []

# Example usage:
# neural_init()
# neural_train(0, XY, 0.001, 50, 0.3, 100, 32)
# result = neural_predict(0, X)
# neural_save('models.txt')
# neural_load('models.txt')



Last edited by TipmyPip; 12/01/23 15:10.

ZorroTraderGPT - https://bit.ly/3Gbsm4S

Moderated by  Petra 

Gamestudio download | chip programmers | Zorro platform | shop | Data Protection Policy

oP group Germany GmbH | Birkenstr. 25-27 | 63549 Ronneburg / Germany | info (at) opgroup.de

Powered by UBB.threads™ PHP Forum Software 7.7.1