Partial shared lengthscale

[adamjstewart]

I'm working with spatiotemporal data, so my input values are 3D: latitude, longitude, and time. Obviously, the lengthscale for space (lat/long) may be very different than the lengthscale for time. Using ard_num_dims=3 will allow me to learn a separate lengthscale for each dimension. However, ideally I would only like to learn 2 lengthscales, one for space and one for time. Is it possible/wise to do this in GPyTorch? The API could look something like ard_num_dims=[0, 0, 1], where the length is the number of dimensions of the input and the number of unique values is the number of lengthscale parameters to learn.

[adamjstewart]

Follow-up question: it's possible to specify constraints on the lengthscale, and it's possible to learn a different lengthscale for each dimension, but is it possible to specify constraints on the lengthscale separately for each dimension?

[adamjstewart]

One possible solution I've come up with: instead of using a single kernel, I could use a combination of 2 kernels like so:

RBFKernel(active_dims=[0, 1]) + RBFKernel(active_dims=[2])

This allows the model to learn a single lengthscale for lat/long and a separate lengthscale for time. It also allows you to specify different constraints for each.

Marking this as "the answer" but let me know if this is not recommended or there is a better approach.

[Balandat]

This works, but the resulting kernel is less expressive since it's an additive structure. Which might be totally fine for your application though. Another alternative that would give you more fine grained control would be to implement your own Kernel with that structure.

[patel-zeel]

A behavior close to ARD would be RBFKernel(active_dims=[0, 1]) * RBFKernel(active_dims=[2]) because ARD is mathematically a multiplication of similar or different kernels applied on each feature.

So, RBFKernel(ard_num_dims=3) is mathematically equivalent to RBFKernel(active_dims=[0]) * RBFKernel(active_dims=[1]) * RBFKernel(active_dims=[2]). I am using word "mathematically" here because computationally RBFKernel(ard_num_dims=3) would be more efficient as discussed in #1806.

[adamjstewart]

I tried this and found some interesting behavior. Using the following kernels:

self.spatial_module = ScaleKernel(RBFKernel(active_dims=[0, 1]))
self.temporal_module = ScaleKernel(RBFKernel(active_dims=[2]))

I found that an additive kernel:

self.covar_module = self.spatial_module + self.temporal_module

allows me to learn a separate outputscale for each kernel, whereas a product kernel:

self.covar_module = self.spatial_module * self.temoral_module

forces both outputscales to be identical. Is this expected? I would expect to be able to learn a different outputscale for each (i.e., spatial trends may be more or less significant than temporal trends).

[patel-zeel]

Programatically thinking, ScaleKernel just facilitates a learnable scalar (positively constrained) multiplied with the kernel. So, in the multiplication case, the resultant scale becomes the scale of the spatial module times the scale of the temporal module. So, their independent values do not have any significance here but only the resultant value of their multiplication.