Questions regarding MAP SAAS model #2909
-
|
Hi all, I am a bit puzzled regarding the map saas model. So far I was just using the botorch/botorch/models/map_saas.py Line 337 in 78de86d I was then treating it for fitting as a usual Line 441 in 78de86d Looking at the implementations, they look quite similar to me, main difference to me is that in the latter one it is treated as an ensemble model, and in the former one as a standard model. Can you give some hints/explanation when to use one? Best, Johannes |
Beta Was this translation helpful? Give feedback.
Replies: 1 comment 4 replies
-
|
They're indeed quite similar. The MAP SAAS models control the sparsity prior of the kernel using a parameter tau. If we use a single model, we have a single tau, which can be a bit too opinionated (though still works well, if you need something cheaper). To balance it out, we can use ensembling with multiple different tau parameters, which is where However, there were some challenges with integrating it into the Modular BoTorch setup in Ax. The legacy Ax generators would let you construct a model using an arbitrary callable but the Modular BoTorch uses a more standardized setup, where it constructs the models from scratch and fits them. This setup didn't support the ensemble model (we've since made it more flexible and I have a prototype that supports the ensemble model), so we tried out the additive kernel instead. Similar to the ensemble model, it also balances out the different sparsity levels, since the kernel sums up kernels with different tau values. There are some differences that make it more prone to over fitting, such as a shared likelihood and joint model fitting, but it is also slightly cheaper (both to fit and within acqf). I think it works pretty well. |
Beta Was this translation helpful? Give feedback.
-
|
Thanks for the explanation, I have now integrated the one with the addtive kernel into our codebase. It works really nice, but based on your experience the ensemble one is even better, but more expensive to fit and more expensive in acqf? |
Beta Was this translation helpful? Give feedback.
-
|
I don't know if I could say one is better than the other. I looked but couldn't find benchmark results comparing the two, but I recall them performing pretty similar overall. The additive kernel is more prone to over-fitting, due to the kernel being over-parameterized. It may not have as well calibrated uncertainty estimates as the ensemble model. But I don't necessarily believe that over-fitting is a bad idea for BO. You over-fit, evaluate a point that you thought was good, learn about it and correct the model in the next iteration. I think it can often be better to have a strongly opinionated model that can take you in some direction than to have an uninformative model with all flat predictions. |
Beta Was this translation helpful? Give feedback.
-
|
Agree! Thanks again for the explanation. I will then stick for now to the additive one ;-) |
Beta Was this translation helpful? Give feedback.
-
|
cc @hvarfner regarding strongly opinionated models in BO |
Beta Was this translation helpful? Give feedback.
They're indeed quite similar. The MAP SAAS models control the sparsity prior of the kernel using a parameter tau. If we use a single model, we have a single tau, which can be a bit too opinionated (though still works well, if you need something cheaper). To balance it out, we can use ensembling with multiple different tau parameters, which is where
get_fitted_map_saas_ensemblecomes from. This is also similar in spirit to the original fully Bayesian model, but avoids costly model fitting with MCMC. It tends to be more robust than the single tau variant, so we would use this model when the fully Bayesian is too expensive.However, there were some challenges with integrating it into the Mod…