[ENH] review distributions extender API with a view to composite distributions

[fkiraly]

Currently, the distribution API seems problematic in cases of composite distributions, such as the TransformedDistribution. We should review and possibly alter, or extend, the extension contract.

Three points of issue:

• internally, methods like _cdf receive 2D np.ndarray in the array case, but cdf expects pd.DataFrame, so a naive internal call needs to attach indices to the np.ndarray again. Calling internal _cdf is not an option due to the boilerplate functions.
• the contract is undefined on what the index and columns of the inner distributions should be, especially when mathematically they are meant to be equal
  • as a sub-problem, what should happen if a user sets index after construction on the outer distribution - should it also set the inner indices?
• _iloc is not nesting well with distributions, as already discussed in [ENH] Default _iloc and _iat to include handling of composite distribution objects #559

[fkiraly]

FYI @joshdunnlime, your opinion would be appreciated!

[joshdunnlime]

• the contract is undefined on what the index and columns of the inner distributions should be, especially when mathematically they are meant to be equal
  • as a sub-problem, what should happen if a user sets index after construction on the outer distribution - should it also set the inner indices?

I would favour setting the inner distribution to be equal to the index and columns of the outer. I have seen couple issues where the mismatch causes errors. It would also help with debugging (you can inspect the inner and easily compare to the outer). Also, if a user plays around with the inner distribution object they will get consistent indices and columns.

Generally, I think a user will change the indices or columns after calling distribution methods like pdf, cdf. I feel it might be over engineering to support this niche case. It should also be fairly clear if they set the index and get an index error or two differing indices in their outputs.

[fkiraly]

I would favour setting the inner distribution to be equal to the index and columns of the outer.

With the current interface, the user has the option to set it in inner and outer. What are we expecting the user to do? Set it in both, or only the outer, or only the inner?

If they set it in only one, should it then override the other? Or should the outer override the inner always?

[joshdunnlime]

My assumptions might be wrong, but is a user likely to initialise the TD directly? It seems to me the majority of the use cases will be to return a TD via predict_proba on the TTR.

If so, I think allowing the outer to set the inner at initialisating is the clear answer. I don't think we need to add complexity by keeping these in sync (at least not now). As I mentioned above, if someone wants to change the index or columns, it is much more intuitive to do so on the dataframe that comes out of a pdf, cdf, etc.

What are the other use cases for the TD or similarly extended distributions?

[fkiraly]

My assumptions might be wrong, but is a user likely to initialise the TD directly?

Unlikely. Still, this issue is more general, about composite distributions, for instance the Mixture, which has the same tricky issue imo.

If so, I think allowing the outer to set the inner at initialisating is the clear answer.

So you would enforce the outer to set index, columns of the inner on construction?

Interesting.

Could you elaborate why you prefer this above the alternatives? For instance:

• have inner set outer
• have inner set outer if only inner present, have outer set inner if only outer present, and some priority or error if both present and discrepant
• do nothing

[fkiraly]

PS: I just released 2.10.0 (because of python 3.14 release and adding quick support).

Have you checked whether the partial work on the TTR is sufficient for your use case? Of course we still want the exact solution with auto-diff, I was just wondering whether the chain of heuristics/defaults was already giving you the features you need (this would indicate it was a good idea to fill in the defaults).

[joshdunnlime]

My assumptions might be wrong, but is a user likely to initialise the TD directly?

Unlikely. Still, this issue is more general, about composite distributions, for instance the Mixture, which has the same tricky issue imo.

I haven't dig into the weeds of the Mixture distributions so I might be missing some thing but I'll try and answer the following questions with that in mind.

Could you elaborate why you prefer this above the alternatives? For instance:

• have inner set outer

For TD I can't see a scenario where this would happen, at least not in the typical workflow of: TTR -> TD. Everything flows from left to right (and from outer to inner), so this design choice would simply be YAGNI w.r.t to the TD.

I assume the typical use case for Mixture is similar with something like:

mixed = Mixture(...)
reg = SomeRegressor(..., distribution=mixed)

If so, I think this still applies but I might be missing something.

• have inner set outer if only inner present, have outer set inner if only outer present, and some priority or error if both present and discrepant

Just YAGNI again. I might be missing something on the mixed use case but if this is the intended use, why add the complexity now?

• do nothing

See my original comment:

I would favour setting the inner distribution to be equal to the index and columns of the outer. I have seen couple issues where the mismatch causes errors. It would also help with debugging (you can inspect the inner and easily compare to the outer). Also, if a user plays around with the inner distribution object they will get consistent indices and columns.

[joshdunnlime]

PS: I just released 2.10.0 (because of python 3.14 release and adding quick support).

Have you checked whether the partial work on the TTR is sufficient for your use case? Of course we still want the exact solution with auto-diff, I was just wondering whether the chain of heuristics/defaults was already giving you the features you need (this would indicate it was a good idea to fill in the defaults).

I've not check it yet as I've been focused on some other stuff but will do. Great to see the quick response to adding 3.14.

[fkiraly]

If so, I think this still applies but I might be missing something.

I think the point where the Mixture interface is directly called, e.g., let's say a user wants to construct a mixture distribution. Or, a developer who wants to create an estimator that returns a mixture distribution. Where should they put index and columns args?

[joshdunnlime]

If so, I think this still applies but I might be missing something.

I think the point where the Mixture interface is directly called, e.g., let's say a user wants to construct a mixture distribution. Or, a developer who wants to create an estimator that returns a mixture distribution. Where should they put index and columns args?

I think in both cases my suggestion works. You would get the following kind of functionality:

norm = Normal()
trunc = Truncated()
mix = Mixture([norm, trunc], index=idx, columns=cols)
s_mix = mix.sample()
s_norm = mix.distributions[0]
s_trunc = mix.distributions[1]

df = s_mix.join(s_norm, rsuffix="_norm").join(s_trunc, rsuffix="_trunc")

For the developer, calling predict_proba and only needing to set the indices once, e.g. Mixture(..., index=idx, columns=cols) is nice.

Of course, adding support for a change to inner to set the outer might be nice, but I'd consider it as just bonus.