Interface for MAP/MLE/VI

[yebai]

Proposal for a unified interface

optimise(model, MAP(), ...)
optimise(model, MLE(), ...)
optimise(model, VI(variational_distribution=q, ...), ...)

This resembles the sample(model, sampler, ...) interface for Monte Carlo sampling-based inference algorithms.

[yebai]

Relates to #2506; cc @Red-Portal

[Red-Portal]

I think one downside would be that referring to the documentation becomes a little tricky since everything requires specialization. For instance, if I only care about VI, doing ?optimise shows the documentation for unrelated functionalities.

[mhauru]

How would the other arguments to optimise look? I'm not familiar with what sorts of arguments VI needs, but if they are different from what MAP/MLE needs, then that would be an argument in favour of separate functions. Another thing that I wonder about is the return value, if that would be very different (point-estimate vs posterior approximation).

I should also check what other PPLs do for this.

[Red-Portal]

@mhauru I think that could be accommodated by pushing everything VI or ML/MAP specific to the object representing the algorithm like vi( ... ). So, the general interface for optimise could be left simple. But this also raises the question of what's the point of making everything go through optimise.

[DominiqueMakowski]

For what it's worth (i.e., from my basic user perspective), I really hope there is some kind of unified API (and it might emerge through some other packages acting like an interface to Turing, but might as well think of it early) à-la-rstan (brms / rstanarm):

brm(y ~ x,, data = df, algorithm = "sampling")

The algorithm can be swapped for anything ("meanfield", "fullrank", and "pathfinder") through global options. This makes is super convenient to prototype a script, test it and make sure it runs using "fast" algorithms, and then just change the default algo for MCMC and run it without any edits needed to the script.

In the case described above there is some additional considerations if one wants to incorporate approaches that return point-estimates (MLE/MAP optimization), but I don't think it poses that big of a challenge

[Red-Portal]

Hi Dominique. I think here, the situation is slightly different since the unified interface is over ML and MAP. And the potential users of ML and MAP are most likely very different from VI. (On the other hand, I think choosing between MCMC, Pathfinder, or VI, for example, is a more likely scenario.) Therefore, I think there is less incentive the unify the interface.

[yebai]

A unified interface for all optimisation-based approximate inference algorithms (MAP/MLE/VI) would be nice. There are some implementation details on the optimise(model, InferAlgo(...), ...) function signature, but that could be figured out.

[yebai]

A more ambitious goal would be infer(model, data = df, algorithm = ...; kwargs):

1. unify sampling, optimisation, and VI-based inference algorithms
2. automatically construct a conditioned model, i.e. perform model() | data under the hood

[SamuelBrand1]

Nice!

Can I also give a shout out for Batch based (e.g. https://turinglang.org/DynamicPPL.jl/stable/api/#DynamicPPL.MiniBatchContext ) and (nearly equivalently) tempered (e.g. #2524)

[Red-Portal]

Hi @SamuelBrand1 , yes, minibatching is on our agenda.

[bvdmitri]

@yebai

A more ambitious goal would be infer(model, data = df, algorithm = ...; kwargs):

this is how RxInfer does it too

[DominiqueMakowski]

_{Imagine a future where models are written in a universal backend-agnostic way (e.g., as a string) and then infer(model_as_str, ..., engine="Turing|RxInfer") ❤}