Modeling several tasks and their derivatives with LMC

[julienmartinelli]

[I opened this originally as an issue only to discover after this discussion section, my problems seems to be a better fit for this rather than an issue]

Hi, I have several tasks f_1,...,f_d and I wish to model them with a linear model of coregionalization (more precisely, SLFM).
Now, let's assume that I also want to add to the joint model the derivatives of each function. Working out the kernel on paper is straightforward given that f_1,...,f_d follow LMC. It is less clear on the code.

Basically, I only know that there is gpytorch.variational.LMCVariationalStrategy as well as gpytorch.kernels.rbf_kernel_grad. what would be the necessary steps to couple these two together, or possible existing models to get inspiration from?

As an additional question, I also want to be able to have different training inputs for any output: that is f_1,...,f_d but also any derivative, they all can have different training inputs.
In the simple case where d=1, can somebody provide a snippet of how to do that? For instance, extending https://docs.gpytorch.ai/en/stable/examples/08_Advanced_Usage/Simple_GP_Regression_Derivative_Information_1d.html to handle task-specific inputs would be enough.
I have been trying to make this work using Hadamard Multitask GP or BoTorch multi task gp models but got unlucky.

[gpleiss]

Copying over the comment from #2210 : @SebastianAment any ideas here?