qNEI at high q outperforming qNEI at low q

[EvanClaes]

Hi all,

I've been playing around with qNEI for bioprocess optimization. I have two in silico bioprocesses which are called mAb and rAAV. For q = 1, 2, 5 the behaviour is as expected (decreasing performance). However, for both cases, qNEI at q = 10 is outperforming all of the former.

See these figures. I'm initialising with sobol sequence of 3, adding 10% Gaussian noise to observations, and doing 100 repetitions. The confidence bands indicate the 10th & 90th percentile.

If it was just for one case I'd blame it on statistical variability, but since I observe it for both cases, maybe something else is going on? The only other explanation I can think of is that at higher q, the additional queries become a bit more 'random' and thus more explorative which might be beneficial for these cases?

[eytan]

I am not sure if it will actually outperform q=1 in all cases, but there is some work that shows how generating a q batch of points and selecting a single point at random is an effective non-myopic acquisition function. You can read a little more on the theory in that case here: http://proceedings.mlr.press/v119/jiang20b/jiang20b.pdf

…

On Sun, Nov 20, 2022 at 10:04 AM EvanClaes ***@***.***> wrote: Hi all, I've been playing around with qNEI for bioprocess optimization. I have two in silico bioprocesses which are called mAb and rAAV. For q = 1, 2, 5 the behaviour is as expected (decreasing performance). However, for both cases, qNEI at q = 10 is outperforming all of the former. See these figures <https://imgur.com/a/19dzqQy>. I'm initialising with sobol sequence of 3, adding 10% Gaussian noise to observations, and doing 100 repetitions. If it was just for one case I'd blame it on statistical variability, but since I observe it for both cases, maybe something else is going on? The only other explanation I can think of is that at higher q, the additional queries become a bit more 'random' and thus more explorative which might be beneficial for these cases? — Reply to this email directly, view it on GitHub <#1510>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAAW34IJGK672ATY63MGX2TWJI4WDANCNFSM6AAAAAASF4J7GY> . You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>

[EvanClaes]

Very interesting, thanks for the referral. I guess that makes sense from an intuitive standpoint, as q(N)EI is also somewhat non-myopic within a single batch?

Enjoy your sunday!

[Balandat]

From the figures it looks like there is still a good bit of variance in the estimate, not sure I'd be confident to say that q=10 works better here. That said I think your intuition here is good, as you crank up q the batch will naturally get more exploratory. Whether that will help with optimization performance will also depend on the quality of the models - if your model is is wrong but highly confident then you're in trouble when using a more exploitative acquisition function. So my suggestion here would be to check the quality of the surrogate models (e.g. by running cross validation and seeing how those statistics evolve over the optimization loop).

[EvanClaes]

Hello Max,

Thanks for the suggestion. I guess that because of the small SOBOL initialisation batch, a confident but inaccurate model in the first qNEI step is definitely a possibility? Will look into it...

Enjoy your sunday!