Sequential candidate generation and set_X_pending

[CihanSoylu]

Hi,

I am trying to understand how exactly the sequential candidate generation work. The setting I was thinking is where the domain is discrete so the appropriate optimization function is optimize_acqf_discrete and say we are using qUpperConfidenceBound with q > 1. Based on the code, the optimize_acqf_discrete function generates a candidate and concatenates this candidate to the X_pending attribute of the acq_function via set_X_pending method. So I try to follow the code and see what does this change for the next candidate. In the code for the qUpperConfidenceBound, the forward method has the decorator @concatenate_pending_points which concatenates the X_pending to the input before calling the forward method. And the forward method calls the model.posterior on its input which passes the input to the model in the eval mode.

1. Here is my confusion: The first candidate was part of the choices and after the selection, it is added to X_pending and removed from the choices (if unique). In the next iteration though, the X_pending is concatenated to the input (choices) in the acq_function (by the decorator). At this point, my expectation was that the new candidate will be the same as the first one. Here is a toy example showing otherwise, that manually sets X_pending

import torch
from botorch.models import SingleTaskGP
from botorch.fit import fit_gpytorch_model
from botorch.acquisition import qUpperConfidenceBound
from botorch.optim import optimize_acqf_discrete

import math

def f(x):
    return torch.sin(x * 2 * (math.pi))

torch.random.manual_seed(seed = 1)
domain = torch.rand(1000,1)
train_x = torch.rand(3,1)
train_y = f(train_x)

gp = SingleTaskGP(train_x, train_y)

qUCB = qUpperConfidenceBound(gp, beta = 1)

# Get a single candidate
candidate, _ = optimize_acqf_discrete(qUCB, choices = domain, q = 1)
print(candidate)
# tensor([[0.0037]])

# Set X_pending with the candidate
qUCB.set_X_pending(candidate)

new_candidate, _ = optimize_acqf_discrete(qUCB, choices = domain, q = 1)
print(new_candidate)
# tensor([[0.9923]])

So clearly the model is somehow "conditioned" on the first candidate by setting the X_pending. My question is where in the code this conditioning is exactly happening?

2. As I was looking into the code, I was expecting to see a call to get_fantasy_model method of the GPyTorch model providing the model predictions for the candidate as the target. Would this be reasonable approach to generate candidates sequentially?

Thank you!

[Balandat]

So the point here is that qUCB (all MC acquisition functions, actually) computes the acquisition value of the concatenated input X_aug = [X, X_pending] (this is automated via the decorator), but the optimization only happens over the input X. Therefore, if a previous candidate X0 is part of X_pending, the acquisition value of acq_(X0) = acq(X_aug) = acq([X0, X0]) is going to be lower than for other values b/c of the repeated occurrence of X0. You can think of this as the conditioning happening implicitly. See Section 5.2 of https://proceedings.neurips.cc/paper/2020/file/f5b1b89d98b7286673128a5fb112cb9a-Paper.pdf for a discussion.

As I was looking into the code, I was expecting to see a call to get_fantasy_model method of the GPyTorch model providing the model predictions for the candidate as the target. Would this be reasonable approach to generate candidates sequentially?

Yes, this would also be a reasonable approach to generating candidates sequentially. But the thing we do is easier / more elegant / faster :)

[CihanSoylu]

Thanks a lot for your response. Unfortunately it is not obvious to me why the repeated occurrence of X0 is causing the acquisition value to be lower than other points? And as a follow up to that, does this affect the acquisition values of the points in the local neighborhood of X0? The reason I am asking this is that, in terms of exploration, for an acquisition function which takes the uncertainty into account, I would want a candidate would reduce the uncertainty of its local neighborhood, which yields a diverse set of candidates. I guess this is not a problem with joint optimization but in the discrete setting, only the sequential optimization is available.

[Balandat]

Think of X_pending as the points at which we have already committed to evaluate the function. The marginal gain of evaluating the function at second time at a point we've already committed to is going to be a lot lower than if it were the first evaluation at that point. You are right that this is not just true for the exact same points, but also for points in the neighborhood. This is the mechanism by which pending points encourage exploration in other areas away from the pending points. This is true whether or not we're in the discrete setting. The reason why optimize_acqf_discrete (or also the mixed version) defaults to sequential optimization (with proper conditioning via pending points) is that the size of the decision set for joint optimization explores quite quickly in the number of points (if the discrete space has cardinality K then joint optimization over a batch of q candidates needs to optimize over a set of cardinality K^q.