Single objective BO assignment

[Amanichab]

A key issue I encountered is related to non-determinism in Ax’s optimization process. I initially used the set_seeds() function to fix NumPy and Torch seeds, but this has no effect on AxClient, which manages its own random state internally.

To achieve reproducible results, it is necessary to explicitly pass the seed to the AxClient constructor:

utils.py

def set_seeds(seed: int = 42):
    np.random.seed(seed)
    torch.manual_seed(seed)
    return seed

sobo_assignment.py

random_seed = set_seeds()
ax_client = AxClient(random_seed=random_seed)

Without this, results can vary across runs, even if the rest of the environment is fixed.
Suggestion: Either add a note in the README to explicitly recommend seed=42, or modify the code to enforce this pattern by default.
Before the changes :
Link (utils.py) : https://github.com/ACC-DataScience/1-single-objective-bo-Amanichab/blob/bab75fd7bc7c4cc4fbeb3844ea58b4ac3a2b54d4/utils.py#L5-L9
Link(sobo_assignment.py) : https://github.com/ACC-DataScience/1-single-objective-bo-Amanichab/blob/bab75fd7bc7c4cc4fbeb3844ea58b4ac3a2b54d4/sobo_assignment.py#L13

After :
Link (utils.py) :https://github.com/ACC-DataScience/1-single-objective-bo-Amanichab/blob/18df92e8fdf7d1704cbef597ca9d7e9e552610a8/utils.py#L5-L11
Link(sobo_assignment.py) : https://github.com/ACC-DataScience/1-single-objective-bo-Amanichab/blob/18df92e8fdf7d1704cbef597ca9d7e9e552610a8/sobo_assignment.py#L21-L31

One of the tests involved computing the deviation . Initially, I used a different (but valid) approach, which gave reasonable results but did not pass the test.

Upon inspecting the test file, I noticed the import of ObservationFeatures, which led me to adjust the method accordingly. Using ObservationFeatures was necessary to match the expected output.
Link : https://github.com/ACC-DataScience/1-single-objective-bo-Amanichab/blob/18df92e8fdf7d1704cbef597ca9d7e9e552610a8/sobo_assignment.py#L164-L194

I'm still troubleshooting the RMSE mismatch. Even with the seed properly set. I’ll continue investigating.

I used the following course-provided functions in my implementation:

• get_feature_importance
• cross_validate
• compute_diagnostics
  Link : https://github.com/ACC-DataScience/1-single-objective-bo-Amanichab/blob/18df92e8fdf7d1704cbef597ca9d7e9e552610a8/sobo_assignment.py#L95-L158

While these were essential, I had to look into their internal code to understand how to use them correctly. They are not documented in the README or elsewhere.It would be helpful to integrate clearer documentation or at least brief usage examples into the course materials, especially since these functions are required to complete the tasks. Ax documentation link in the README is broken . Here is the correct one:
https://ax.readthedocs.io/en/0.5.0/modelbridge.html#model-bridges

A few quick comments about the course, not critical, but worth pointing out:

• For the Single Objective Assignment, in the test file sobol_asssignment_test.py, the test expects a file named sobol_asssignment_ans.py, but in the Codespaces environment, the file is named sobol_assignment.py . And there was a small typo, that took me a while to catch, which was a bit frustrating.(the triple "s").
  Link (before): https://github.com/ACC-DataScience/1-single-objective-bo-Amanichab/blob/bab75fd7bc7c4cc4fbeb3844ea58b4ac3a2b54d4/sobo_assignment_test.py#L14
  Link(after) : https://github.com/ACC-DataScience/1-single-objective-bo-Amanichab/blob/18df92e8fdf7d1704cbef597ca9d7e9e552610a8/sobo_assignment_test.py#L12-L16
• Similarly, for the Single Objective test, the evaluation of the gp_improvement metric was tested but not mentioned in the README. (That might be on me, but I don’t recall seeing it explicitly.)
  Link : https://github.com/ACC-DataScience/1-single-objective-bo-Amanichab/blob/bab75fd7bc7c4cc4fbeb3844ea58b4ac3a2b54d4/sobo_assignment_test.py#L88-L95
  These aren’t major issues, but flagging them could help reduce confusion for future students.

[sgbaird]

Thanks, @Amanichab! Could you include links to the specific locations? (you can edit your original post). It's fine if the links are to your private assignment repo - I have access and can see them.