5x5 Cross Validation

[jackhhewitt]

Hi,

I'm wanting to use a 5x5 CV strategy so I can compare best_trial fold metrics with some other models that also use 5x5 CV.

I tried implementing a custom class for RepeatedStratifiedKFold in splitter.py. At runtime I can see from the study.trials_dataframe that there are 25 (5x5) metrics per fold which is fine. The issue I'm facing is during build_best and that the RepeatedStratifiedKFold isn't recognised as part of AnyCvSplitter during deserialisation.

This is the ValidationError message:

Whats the fix? Or is there another method of utilising repeated 5x5 CV in the Settings part of the config?

Cell In[107], [line 3](vscode-notebook-cell:?execution_count=107&line=3)
      1 study = optimize(config, study_name='study')
----> [3](vscode-notebook-cell:?execution_count=107&line=3) build_best(buildconfig_best(study), r"M:\PGR1\QSAR\final_models\X60\tobramycin\5x5\tm60_5x5.pkl")

File c:\Users\sgjhewi2\.conda\envs\Q313\lib\site-packages\optunaz\three_step_opt_build_merge.py:264, in buildconfig_best(study)
    262 def buildconfig_best(study):
    263     try:
--> [264](file:///C:/Users/sgjhewi2/.conda/envs/Q313/lib/site-packages/optunaz/three_step_opt_build_merge.py:264)         return buildconfig_from_trial(study, study.best_trial)
    265     except RuntimeError:
    266         return buildconfig_from_trial(study, study.best_trials[0])

File c:\Users\sgjhewi2\.conda\envs\Q313\lib\site-packages\optunaz\config\build_from_opt.py:48, in buildconfig_from_trial(study, trial)
     44 if optconfig_json is None:
     45     raise ValueError(
     46         "Study does not have a user attribute with Optimization Configuration."
     47     )
---> [48](file:///C:/Users/sgjhewi2/.conda/envs/Q313/lib/site-packages/optunaz/config/build_from_opt.py:48) optconfig = deserialize(OptimizationConfig, optconfig_json)
     50 trial = remove_algo_hash(trial)
     51 descriptor_json = trial.params[TrialParams.DESCRIPTOR]

File c:\Users\sgjhewi2\.conda\envs\Q313\lib\site-packages\apischema\utils.py:400, in deprecate_kwargs.<locals>.decorator.<locals>.wrapper(*args, **kwargs)
    398         if replacement:
    399             kwargs[replacement] = kwargs.get(replacement, arg)
...
    704 elif field_errors or errors:
--> [705](file:///C:/Users/sgjhewi2/.conda/envs/Q313/lib/site-packages/apischema/deserialization/methods.py:705)     raise ValidationError(errors or [], field_errors or {})
    706 return self.constructor.construct(values)

ValidationError: ValidationError: [{'loc': ['settings', 'cv_split_strategy', 'name'], 'err': "not one of ['Stratified'] (oneOf)"}, {'loc': ['settings', 'cv_split_strategy', 'name'], 'err': "not one of ['Random'] `(oneOf)"}]