Bayesian Optimization and Active Learning Cookbook

[slishak-PX]

At PhysicsX, we build AI to improve the design, manufacturing, and operation of complex products and processes. We've found BoTorch to be a useful component in many of our projects, so wanted to give something back to the community.

Last year, we decided to put together a practical guide to getting started with BoTorch, for internal use by our Machine Learning Engineers and Data Scientists. Today, we've published this guide for all to see:

Bayesian Optimization and Active Learning Cookbook

We think that it works quite well as a beginner's guide to a range of basic optimization tasks in BoTorch. We'd love to get some feedback on the content, so please write below if anything seems off. We know there are some issues with how the plots are displayed on some devices, and will be working on this.

The original form of this document is a Jupyter Notebook, and if there's interest, we'd be happy to share a more "pure" version of it - although for now it won't be in a completely runnable state as we can't currently share all of the plotting code and the library we built on top of @peterdsharpe's excellent AeroSandbox,

We hope that this resource introduces engineers in industry to the potential of Bayesian optimization, and numerical optimization in general. For some further context, we describe our perspective on the current state of optimization in engineering in a previous related post: Engineered to Fail: You're Doing Optimization Wrong.

[Balandat]

This is awesome! Love the cookbook and the visualizations & animations. This will be super helpful for the non-expert to understand what's going on and develop some intuition. Overall this looks great (some minor points of feedback below). I think it would be great to link this in our readme / docs as a resource for people to get a walkthrough, assuming you are ok with that.

One thing I'm very curious about is why you are using BoTorch over Ax (https://github.com/facebook/Ax), which is a higher-level library with a more user-friendly interface that doesn't require as verbose a problem setup an is also maintained by our team. Are you doing enough customization of the models / acquisition functions / objectives that you've found that you need the additional flexibility that BoTorch offers? If you have considered Ax, I'd love to get some feedback from you about pain points you've found with it for your work so we can make it better!

A couple of minor points of feedback:

1. For the comparison of acquisition functions, you're showing a single run (which makes sense as it maps to the visualization in the respective tab). But it could give people the wrong impression that this generalizes. Maybe just add a comment that there is randomness in the optimization and that for assessing average/expected performance you'd want to run multiple replicates to understand the distribution of performance (and probably also point out that how an acquisition function performs will depend somewhat on the nature of the underlying problem as well).
2. You're using EI to illustrate things, which is probably a good entry point for people new to BO. But we've found log(N)EI - a drop-in replacement for EI - to perform a lot better in many cases, so it would be good to either use that or at least point the reader to it. [Edit: I see you're using it further below - probably worth commenting on why when you introduce it].
3. For parallelism, you talk about batch acquisition functions. In practice one often uses asynchronous parallelism where one generates a new candidate to evaluate whenever some new resources for evaluation become available (based on all the data available up to that point). It would be good to point that out as well and discuss how this can be handled with providing "pending points" to the acquisition function.
4. "And we'll set some defaults for the optimization parameters" <- probably good to point the reader to the docs/docstring for this in case they're interested to dig deeper o/w they'll be confused what these are.
5. Some of the plots have overlapping axis labels which makes them hard to read.
6. "qLogNoisyExpectedHypervolumeImprovement, which is sometimes abbreviated to qNEHVI." <- this would be qLogNEHVI
7. The Pareto front animation is super cool! I'd recommend cutting off a bit more of the y axis above the reference point to make the tradeoffs on the actual front easier to see.
8. In the context of qNegIntegratedPosteriorVariance: "Note that this process is slow due to the extra evaluations required, so we'll run it for a relatively low number of iterations." <- maybe clarify that "evaluations" here pertains to evaluating the surrogate model, and not the underlying black box function.
9. Love the BO vs. GA comparison as a function of the simulation time! We often work on problems where a single evaluation takes many hours, days, sometimes even weeks. Could be worth commenting on that as well as using GAs in such a setting is pretty much infeasible.

[Balandat]

The original form of this document is a Jupyter Notebook, and if there's interest, we'd be happy to share a more "pure" version of it.

Having the source notebook would be great as well.

although for now it won't be in a completely runnable state as we can't currently share all of the plotting code and the library we built on top of @peterdsharpe's excellent AeroSandbox,

Understood - when you say "currently" do you expect this to change in the future?

[slishak-PX]

Thank you so much for the really detailed feedback! We'll address all of the points you made over the coming weeks.

To respond to few of your points below:

I think it would be great to link this in our readme / docs as a resource for people to get a walkthrough, assuming you are ok with that.

This would be really awesome to see, thank you for suggesting this!

One thing I'm very curious about is why you are using BoTorch over Ax (https://github.com/facebook/Ax)...

There are a few reasons behind this:

• Internally, we actually have a tool like Ax which provides common abstractions for a variety of optimizers in a way that is compatible with our customer platform (and is what we used to run the BO vs GA comparison at the end of the cookbook). Early on, this was using Ax under the hood to run BO, but it seemed unnecessary to have another layer/dependency in the stack.
• We do a lot of customization too - for example, many of our deep learning surrogate models support UQ, so performing BO using these felt a lot more straightforward to deal with in raw BoTorch (especially if the resulting acquisition function is not differentiable, which is sometimes the case).
• I just find BoTorch more intuitive than Ax - all the moving parts (acquisition functions, models, transforms) are explicitly defined and easy to understand/manipulate by someone who is already familiar with those concepts. Ax which tries to simplify things but simultaneously introduces a new set of jargon to learn, although the glossary is helpful.
• I also love the BoTorch documentation - it's introduced us to quite a few new approaches, but it's not clear how many of these would translate into Ax.

For the comparison of acquisition functions, you're showing a single run (which makes sense as it maps to the visualization in the respective tab)...

This is a great point - I was already thinking that the comparison of "best observation" is a bit misleading, because it might lead a reader to mistakenly think that the acquisition values should be ranked by this (when in fact it's mostly just due to the noise in the objective function).

although for now it won't be in a completely runnable state as we can't currently share all of the plotting code and the library we built on top of @peterdsharpe's excellent AeroSandbox,

Understood - when you say "currently" do you expect this to change in the future?

I don't think there are any insurmountable barriers to this, it just needs a bit more time. The plotting code isn't shared simply because it's messy and I'd want a bit more time to clean it up. The aircraft design library is a bit trickier, because we'd need to extract only the relevant parts of the library. Another aspect is that the whole notebook takes about a day to run on a reasonably beefy machine, so although we would love for it to be runnable to reproduce the results, we didn't think many people would have the patience to do this.