Uncertainty calculation of Deep GP incorrect?

[fweberling]

Hi everyone,

the Deep GP implementation has very weird predicted variances. I used the implementation of the Multitask 2-layer Deep GP tutorial and looked at regions away from the data points and used a simple scaled RBF kernel instead of the Matern kernel besides that I didn't change anything in the code. Interestingly, the variance of a specific test point close to the training points changed when I increased the window size and the resolution in torch.linspace(0,1,51) to torch.linspace(-3,4,200) which cannot be right.

Here, an interval of 200 test points from [0,1] is used. The predicted variance of the test point right in the middle of each image is var[100,:] = [0.1145, 0.1110, 0.1353, 0.1099] where each element corresponds to the specific task:

Here, an interval of 200 test points from [-3, 4] is used. The predicted variance of the same test point right in the middle of each image changes to var[100,:] = [0.6471, 0.8840, 0.7339, 0.5928] where each element corresponds to the specific task. This cannot be right.

When I now consider the same implementation of the Deep GP but now only with one layer the uncertainties also change as previously stated for the test point in the middle of the plot and the uncertainties away from the data point look very weird. They have a wave-like structure.

var[100,:] = tensor([0.0520, 0.0501, 0.0608, 0.0536])

var[100,:] = tensor([1.1507, 1.1499, 1.1616, 1.1552])

Is this possibly a bug? I have not changed the code of the tutorial. I only adjusted the resolution of the test point and the interval in torch.linspace(0,1, 51).

I would love to get your feedback!
Friedrich

[gpleiss]

The output of a DeepGP model is non-deterministic, so it is possible that the different outputs are due to randomness and not due to different model inputs.

[fweberling]

Deep_GPtrial.zip

This is the Jupyter Notebook I used.

[fweberling]

These weird uncertainty predictions of the Deep GP also do not change if I don't use gpytorch.settings.fast_pred_var() . So, I guess it won't be due to the computations at test time using the Lanczos algorithm. Massively increasing the number of inducing points also does not help.

[fweberling]

I have found the issue here:

def predict(self, test_x):
        with torch.no_grad():

            # The output of the model is a multitask MVN, where both the data points
            # and the tasks are jointly distributed
            # To compute the marginal predictive NLL of each data point,
            # we will call `to_data_independent_dist`,
            # which removes the data cross-covariance terms from the distribution.
            preds = model.likelihood(model(test_x)).to_data_independent_dist()

        return preds.mean.mean(0), preds.variance.mean(0)

The predict method in the Deep GP model of the GPyTorch tutorial on Deep GPs does not work correctly. Especially, .to_data_independent_dist() seems to do something wrong. I suspect it is about some reshaping.

Using this method the uncertainties become correct:

def predict(self, test_x):
    with torch.no_grad():

      preds = model.likelihood(model(test_x))

      preds_mean = preds.mean.mean(axis=0)
      preds_var = preds.covariance_matrix.mean(axis=0).diag().reshape(num_tasks, num_tasks * self.train_x_shape[0]).T

    return preds_mean, preds_var

[gpleiss]

Hmm okay this does in fact seem like a bug on our end. Thank you so much for digging into this. Let's make this into an issue.

[gpleiss]

@fweberling this I think should be closed by #2123