how to set number of samples for MCAcquisition

[shalijiang]

Is there a rule-of-thumb to set the number of samples for MCAcquisition function (say qExpectedImprovement? e.g., k*q for some k?

Seems now the default number of 512, but I suppose #samples should increase as q?

[Balandat]

Indeed, the higher-dimensional the integral (dimension is qm, where q is the number of points considered jointly and m is the number of outputs of the model), the higher the number of samples should be if you care about accuracy.

We highly recommend using qMC (Sobol) instead of MC, in which case the number should ideally be 2^k for some k>1 (get some good properties for the underlying low-discrepancy sequences used. For MC you get a CLT with variance O(1/sqrt(N)) for the estimation error, the asymptotic bounds for qMC are much better, depending on regularity properties of the function. 512 Sobol samples is a relatively high default and should work quite well for many cases.

What I suggest you do in practice is to compute the value (fixing the model and parameters) of the acquisition function for a range of MC samples including some large number (e.g. 2^12), and then plot the difference in value as a function of N. That will give you a good idea what number of samples is necessary. You can repeat this for a bunch of x values to see how this varies with the shape of the posterior.

[Balandat]

fwiw I plan to write up some docs on this in the near future