Warped Gaussian Process Regression

[amaissen]

Hi together,

I have already successfully implemented some basic models. Now I want to implement warped Gaussian Processes (wGP), as presented in this paper. The general idea of wGPs is to transform the observed targets (not the features) with a learnable parametric monotonic function f(·), and doing GP-Regression on the transformed targets. However, the GP-Regression and the parametric function f(·) are learned simultaneously.

Well, the negative log likelihood of basic gaussian processes regression with simple noise looks the following:

$NLL = -\log p(y|X, \Theta) = \frac{1}{2}y^{T}C^{-1}y + \frac{1}{2}\log{|C|} + const$, for $C=K(X,X) + \sigma^2*I$

In the paper they make use of the properties of change of variables, to derive the adjusted negative log-likelihood:

$NLL = -\log p(y|X, \Theta) = \frac{1}{2}f(y)^{T}C^{-1}f(y) + \frac{1}{2}\log{|C|} - jacobbi\ term + const$

So my idea was to implement a custom likelihood (e.g. WarpedGaussianLikelihood) that captures this transformation. Then I planned to extend gpytorch.models.ExactGP as usually, however supply a custom prediction_strategy, since we need to transform the input_targets with the learned transformation. Moreover, one could maybe overwrite ExactGP. __call__. to handle the evaluation mode differently, e.g. first calling super().__call__ and then applying the inverse transformation, etc.

Before I try out my ideas, I wanted to make sure if Warping for target variables is not already supported by GPyTroch. Or maybe you have some better ideas how one could implement this feature/model. I would be very thankful for any comments.

[Balandat]

So we have a setup for a related thing in botorch for doing input warping with learnable transforms: https://github.com/pytorch/botorch/blob/main/botorch/models/transforms/input.py#L796. This is not the same thing (it's the inputs not the outputs), but related.

In terms of implementing the output warping, my hunch is that you could use the added_loss_terms functionality on the model that is then automatically added to the base loss in the MLL.

[andrewgordonwilson]

DKL is probably the most obvious and general example of learning an input transformation (not outputs).

[amaissen]

Thank you for the fast reply and the examples for warping features. Despite the tasks are related, warping the targets seems to be a little bit trickier. Because, we also have to apply an inverse transformation to the resulting predictive distribution in the latent space, which is not required when warping features.

Thank you for pointing out the added_loss_terms functionality. I have to check if this will help.