Verification on the GP tuner accuracy when perform HPO/Save GP tuner iteratively

[haison19952013]

Hi, NNI community,

I'm interested in learning more about the Gaussian Process (GP) tuner predictions within the NNI Hyperparameter Optimization (HPO) process.

To achieve this, I'd like to conduct an experiment based on the following pseudocode, which builds upon the existing GP-based HPO:

- Input: x (NN hyperparamtes)
- Output: y (NN accuracy)
1. Construct an initial GP with x = x_initial
2. for i = 0 to max_iteration:
    2.1. Obtain x_next by evaluating an acquisition function
    2.2. Generate the true y(x_next)
    2.3. **(My additional step):** Use the GP to predict y^(x_next)  # For investigating GP prediction accuracy
    2.4. **(My additional step):** Compare y^(x_next) and y(x_next)  # For investigating GP prediction accuracy
    2.5. Construct an adaptive GP with x = x ∪ x_next  # Update GP with new data point

• My question is, are there any currently supported functions within NNI that allow me to perform step 2.3, which involves predicting the target value using the GP? or at least, save the GP iteratively and perform the experiment later
• I believe this exploration can provide valuable insights into the behavior and accuracy of the GP during HPO. Thanks in advance for your assistance!

[haison19952013]

• Solved the problem by manually using the functions inside nni package. The code verification was done on a Forrester toy function:

from nni.algorithms.hpo.gp_tuner import GPTuner, GPClassArgsValidator
import os
from datetime import datetime
import pickle
import matplotlib.pyplot as plt
import numpy as np

search_space = {
    'x': {'_type': 'uniform', '_value': [0,1]}
}

def hf_function(x):
    import numpy as np

    return ((x * 6 - 2) ** 2) * np.sin((x * 6 - 2) * 2)

n_initial = 5
n_max = 20
GP_tuner = GPTuner(optimize_mode="minimize", utility='ei', kappa=5, xi=0, nu=2.5, alpha=1e-6, cold_start_num=n_initial, selection_num_warm_up=100000, selection_num_starting_points=250)
GP_tuner.update_search_space(search_space)
x_acq_list = []
f_acq_list = []
max_acq_list = []
general_error_list = []
# Adaptive modeling

for i in range(n_max + 1):
    print(i)
    x_next_dict, max_acq, x_acq, f_acq = GP_tuner.generate_parameters(parameter_id = i)
    max_acq_list.append(max_acq)
    x_acq_list.append(x_acq)
    f_acq_list.append(f_acq)
    x_next_array = GP_tuner._space.params_to_array(x_next_dict)
    # with open('model.pkl','wb') as f:
    #     pickle.dump(GP_tuner._gp,f)
    y_next = hf_function(x_next_array)[0]
    if max_acq is not np.nan:
        x_ = GP_tuner._space.params[-1].reshape(-1,1)
        y_mean_next, y_std_next = GP_tuner._gp.predict(x_, return_std=True)
        general_error = (-y_mean_next[0] - y_next)**2 + y_std_next[0]**2
        general_error_list.append(general_error)
    GP_tuner.receive_trial_result(parameter_id= i, parameters = x_next_dict, value = y_next)
    

# Plotting for optimization process
x_ff = np.linspace(0, 1, 50).reshape(-1, 1)
y_ff = hf_function(x_ff.reshape(-1, 1))
y_mean, y_std = GP_tuner._gp.predict(x_ff, return_std=True)
x_ = GP_tuner._space.params
y_ = -GP_tuner._space.target

fig, (ax1,ax2) = plt.subplots(nrows=2, ncols=1, sharex=True, figsize=(10,6), height_ratios=(4,2))
ax1.plot(x_ff, y_ff, label='True function')
ax1.plot(x_ff, -y_mean, label='GP mean')
ax1.fill_between(x_ff.flatten(), -y_mean - y_std, -y_mean + y_std, alpha=0.2, label = '95% CI')
ax1.scatter(x_[:n_initial], y_[:n_initial], color='r', label='Initial samples')
ax1.scatter(x_[n_initial:n_max], y_[n_initial:n_max], color='g', label='Adaptive samples')
ax1.scatter(x_[n_max:], y_[n_max:], color='b', label='Next sample', marker='*')
ax1.legend()
ax1.set_ylabel('Objective function')
ax2.plot(x_acq_list[-1], f_acq_list[-1], color='r', label='Acquisition function')
ax2.scatter(x_[-1], max_acq_list[-1], color='b', label='Next sample', marker='*')
ax2.legend()
ax2.set_xlabel('x')
ax2.set_ylabel('Acquisition function')

# Plotting for convergence process
fig, ax = plt.subplots(figsize=(10,3))
ax.plot(range(n_initial + 1, n_max + 2),general_error_list)
ax.set_xlabel('ith samples')
ax.set_ylabel('Generalized error')
ax.set_yscale('log')
ax.set_xticks(list(range(n_initial + 1, n_max + 2)))
plt.show()