How to evaluate and train this model? + dataset problem

[kokhazade]

Hi @arturossouza and @wrfrohlich,

Thank you for sharing your source code.
I am going to evaluate the pretrained model and also train this model on my machine. I downloaded the dataset and as you mentioned in your paper, I put 14 subjects in train directory and the remained subject in validation directory. At this step, which notebook should I run to evaluate the saved model and which one to train the model from the scratch? Is there any sample code to go through evaluation and training process?
I ran the reservoirTesting.ipynb but I faced the below Error:
No such file or directory: 'data/preprocessedData/training/trainingData.pickle'
Where should I access this file?

[arturossouza]

Hi @kokhazade , thanks for reaching out to us.

Answering your questions on the nexts comments

[arturossouza]

At this step, which notebook should I run to evaluate the saved model and which one to train the model from the scratch? Is there any sample code to go through evaluation and training process?

The first step of our model is the preprocessing, therefore, you should run this notebook: main/01-preprocessing.ipynb

Note that the last cell of this notebook is

setPreprocessingCheckpoint(moStressPreprocessing)

This will create a checkpoint folder on your local machine in order to avoid do the preprocessing again.

At this point, you can use any notebook inside of the main folder with the exception of the nbeatsFeatureExtractor, and that is why we are current developing this code on a separeted repository and the code on this notebook don't work very well.

But please, use the other notebooks, I belive that they can run just fine.

The training is executed via the execute method inside of MoStressNeuralNetwork class, and in all experiments is called like this.

evaluator = EvaluateModel(
    { "features": validationData["features"], "targets": validationData["targets"] },
    moStressNeuralNetwork.modelFullName,
    moStressNeuralNetwork.model,
)
evaluator.executeEvaluation()

And you can see both calls on the end of each notebook of models

The evaluation runs similarly with the method executeEvaluation inside of the EvaluateModel class and goes like this:

moStressNeuralNetwork = MoStressNeuralNetwork(moStressConfigs, dataset)
moStressNeuralNetwork.execute()

At the end of all models notebook we have the calls for this methods

[arturossouza]

I ran the reservoirTesting.ipynb but I faced the below Error:
No such file or directory: 'data/preprocessedData/training/trainingData.pickle'
Where should I access this file?

I have the following folder structure on my local machine

root
--data
-----preprocessedData
-------tranining
----------traniningData.pickle
-------validation
----------validationData.pickle

This structure is created by the setPreprocessingCheckpoint() function on utils/preprocessingCheckpoint.py, which is called after the main/01-preprocessing.ipynb is executed.

Therefore, to run the reservoirTesting.ipynb and other notebooks, please, run the preprocessing notebook first

[arturossouza]

Last comment: when we created this code we were more concern about the models and those stuffs, therefore, some of the general functions as setPreprocessingCheckpoint() function on utils/preprocessingCheckpoint.py may not work really well in all local machines because we probably adjusted the function to work only in our local machine.

That’s been said, if you want, please, create a pull request to our repository to fix anithing you find and we will be happy to review your PR.

[kokhazade]

Dear @arturossouza,
Thank you so much for the detailed information.
I will try to train and evaluate it by considering your comments and ping you back in case of any problem.