Issue in combining multitask GP and sparse GP

[GHU2021]

Hi,

I want to implement multitask sparse gaussian processes. I defined model as follows:
class MultitaskSparseGPModel(gpytorch.models.ExactGP):
def init(self, train_x, train_y, likelihood):
super(MultitaskSparseGPModel, self).init(train_x, train_y, likelihood)
self.mean_module = gpytorch.means.MultitaskMean(gpytorch.means.ConstantMean(), num_tasks=24)
self.base_covar_module = gpytorch.kernels.LinearKernel()
self.covar_module = gpytorch.kernels.MultitaskKernel(
gpytorch.kernels.InducingPointKernel(self.base_covar_module, inducing_points=train_x[train_x_indexes,:], likelihood=likelihood), num_tasks=24)

def forward(self, x):
    mean_x = self.mean_module(x)
    covar_x = self.covar_module(x)
    return gpytorch.distributions.MultitaskMultivariateNormal(mean_x, covar_x)

Also, here is some settings in my program.
train_x_indexes = range(0,716,7)
likelihood = gpytorch.likelihoods.MultitaskGaussianLikelihood(num_tasks=24)
model = MultitaskSparseGPModel(train_x, train_y, likelihood)

train_x is a tensor of size 716x168, train_y is a tensor of size 716x24, test_x is a tensor of size 358x168, and test_y is a tensor of size of 358x24.

I got the following error message when the program wanted to calculate loss (i.e., loss = -mll(output, train_y)):

File "..\multitask_gaussian_likelihood.py", line 117, in _shaped_noise_covar
eye_lt = ConstantDiagLazyTensor(torch.ones(*shape[:-2], 1, dtype=dtype, device=device), diag_shape=shape[-2])
IndexError: tuple index out of range

I appreciate your help.

[gpleiss]

train_x_indices is supposed to map each input to a task. You have 24 tasks, so the values of train_x_indices should be between 0 and 23.

Update I misunderstood the question. New response below.

[GHU2021]

Thank you Dr. Pleiss for provided information.

Since I am using Multitask GP Regression (https://docs.gpytorch.ai/en/stable/examples/03_Multitask_Exact_GPs/Multitask_GP_Regression.html), all tasks use the same training data set. I use one training data for all 24 tasks.

Based on Titsias’ paper and some other papers, in Sparse Gaussian Process Regression (SGPR) example (https://docs.gpytorch.ai/en/stable/examples/02_Scalable_Exact_GPs/SGPR_Regression_CUDA.html), a subset of training data (i.e., some of samples of training data) are used as inducing points. For example, in SGPR example, inducing_points=train_x[:500, :] are 500 samples out of 13279 samples in train_x.

My program works fine with SGPR model. Also, my programs works fine when I use MTGP. To extend SGPR to multitask SGPR, as you suggested (if I understood and applied correctly), I considered InducingPointKernel inside MultitaskKernel, defined linear kernel as a base kernel, and provided indices of training data using train_x_indices.

As you suggested, I changed the values of train_x_indices to the values between 0 and 23 but again I got the same error message.

I am confused why I should use values between 0 and 23 instead of 0 to 716 (I have 716 samples in my training data).

Please correct me if I am wrong in any of my explanations and understandings.

Thank you very much

[gpleiss]

The train_x_index should assign each data point to a task index. Imagine that you have 4 different tasks, and 10 data points. train_x_index = torch.LongTensor([0, 3, 1, 1, 2, 3, 3, 0, 0, 1]) would imply that the first data point belongs to task 0, the second data point belongs to task 3, the third data point belongs to task 1, the fourth data point also belongs to task 1, and so on...

Update I misunderstood the problem. New response below.

[gpleiss]

Your setup looks correct. Based on the error that you are getting...

File "..\multitask_gaussian_likelihood.py", line 117, in _shaped_noise_covar
eye_lt = ConstantDiagLazyTensor(torch.ones(*shape[:-2], 1, dtype=dtype, device=device), diag_shape=shape[-2])
IndexError: tuple index out of range

... it seems likely that you are using the incorrect likelihood. Are you using gpytorch.likelihoods.GaussianLikelihood, or gpytorch.likelihoods.MultitaskGaussianLikelihood?

[GHU2021]

Thank you Dr. Pleiss for your response.

I used the following command:
likelihood = gpytorch.likelihoods.MultitaskGaussianLikelihood(num_tasks=24)

[gpleiss]

Hmm... this may be an issue on our end. Can you supply a fully-runable code example that reproduces the error message?

[GHU2021]

Here is the code example:

import math
import torch
import gpytorch

train_x = torch.linspace(0, 1, 100)

train_y = torch.stack([
torch.sin(train_x * (2 * math.pi)) + torch.randn(train_x.size()) * 0.2,
torch.cos(train_x * (2 * math.pi)) + torch.randn(train_x.size()) * 0.2,
], -1)

class MultitaskGPModel(gpytorch.models.ExactGP):
def init(self, train_x, train_y, likelihood):
super(MultitaskGPModel, self).init(train_x, train_y, likelihood)
self.mean_module = gpytorch.means.MultitaskMean(gpytorch.means.ConstantMean(), num_tasks=2)
self.covar_module = gpytorch.kernels.MultitaskKernel(gpytorch.kernels.InducingPointKernel(gpytorch.kernels.LinearKernel(), inducing_points=train_x[range(0,100,10)], likelihood=likelihood), num_tasks=2, rank=1)
def forward(self, x):
mean_x = self.mean_module(x)
covar_x = self.covar_module(x)
return gpytorch.distributions.MultitaskMultivariateNormal(mean_x, covar_x)

likelihood = gpytorch.likelihoods.MultitaskGaussianLikelihood(num_tasks=2)
model = MultitaskGPModel(train_x, train_y, likelihood)

model.train()
likelihood.train()

optimizer = torch.optim.Adam(model.parameters(), lr=0.1) # Includes GaussianLikelihood parameters
mll = gpytorch.mlls.ExactMarginalLogLikelihood(likelihood, model)

optimizer.zero_grad()
output = model(train_x)
loss = -mll(output, train_y)

[gpleiss]

PR #2121 should address this issue.