Informative priors

[DominiqueMakowski]

Now that I think of it @strengejacke, your tutorial paper about informative priors could be boosted by the implementation of an easy-to-use function (let's call it informative_priors) that would facilitate the set-up of informative priors.

Formatted priors object

I believe that most researchers would be interested in using more informative priors, if only that was simpler. One of the function of such function could be to provide a unified interface to prior specification, that would return a prior object compatible with what they want to use, i.e., brms, rstanarm etc.

Standardized informative priors based on interpreted categorical sizes

Of course, our initial reaction is that informative priors have to be decided on a case by case basis, in regards to prior literature and expectations. While this is, of course, true, one could potentially imagine a second approach to the problem.

For instance, it could be interesting to allow for a more naturalistic prior specification. Similar to what sometimes we think, and what we'd like to say, e.g. "I expect this effect to be small". Now how to transform "small" into something mathematically acceptable? We could refer to an interpretation grid (we could transfer some utilities for that from report). Let's say that a standardized coef for "small" is 0.2. We could unstandardize this value based on the mean/SD of the data, that would give us the location of a prior corresponding to a "small" effect. The scale could be automatically adjusted accordingly.

E.g. informative_priors_rstanarm(y ~ A + B, size =c("small", "medium"), data = df) would return a list with priors set that one could then just pass to the rstanarm model.

Believers vs. Agnostic vs. Sceptic priors

Finally, an argument could control the "type" of prior. Three of them make particular sense to me. First of all, the agnostic prior is similar to the one set by default, for instance in rstanarm (although I find its automatic scaling to be way too generous), i.e., one located at 0. This one is suited for true exploratory analysis. On the contrary, a believer's prior can be useful, for instance in small samples or noisy data, would be located at the expected effect value. If we take the example above, and I expect the effect to be "small", I would place a believer's prior at 0.2 (or its unstandardized equivalent). However, in large datasets or for very robust data, one might want to set sceptic priors, that would be located at the same location than a believer's, but on the opposite direction. A sceptic prior for the example above would be set at -0.2. Such feature could be easily added to our function.

informative_priors_brms(y ~ A, size ="medium", data = df, type = "sceptic")

[strengejacke]

ok, got it. sounds like a good idea! might be some hassle for more complex brms-models, though, probably it's safer to pass a model instead of a formula, or we can check if insight can do the work for us.

[DominiqueMakowski]

This needs:

• implementation of unstandardize() in effectsize, that would take standardized parameters and convert them back to a raw scale (based on the SD/MAD and mean/median of the response variable).
• informative_priors() that would use unstandardize to figure out the priors based on a formula and return it in a stan/compatible format.

We could write an example illustrating a full real-data "study" (the master tutorial?) (showing all facets of bayestestR):

1. We have a question (the relationship between x and y)
2. We make a general hypothesis/prediction (x is positively related to y)
3. We operationalize this hypothesis:
   • Come up with the model information (type of model, formula etc.)
   • Come up with operationalized predictions (I expect a small and positive coefficient)
4. We get the data
5. Prior generation (informative_priors(y ~ x, size="small", direction="positive", data=df)). This returns 3 priors (agnostic, believer and skeptic).
6. We fit the believer's model.
7. Is it a good model? Model evaluation (performance, posterior checks etc.)
8. Yes, -> Interpretation of results (the parameters).
9. We fit the agnostic and skeptics. Does adopting a different perspective significantly change our initial interpretation? "No, the results are consistent"
10. Conclude

[mattansb]

For factors we would need to incorporate contr.bayes, and make sure the scale of prior of continuous is half that of the factor (just like how you can get Beta coefs with 2sd).

[DominiqueMakowski]

nice, we could add that indeed, so this "walk-through" would be fairly complete.

[strengejacke]

reminder to myself: continue working on this paper...

[DominiqueMakowski]

aha! I found your demon, now I will bump this issue just as you bump the news-file one 😈

[DominiqueMakowski]

Looking back at this, this really would be a nice thing to have 🤩

[strengejacke]

Yes yes, it's on its way, and you'll be updated. 🙂

[DominiqueMakowski]

haha nah but I was referring to the bayestestR function to make stan-compatible priors objects

[strengejacke]

Yeah, but that could be part of the work on the paper, so we might introduce that function there...

[DominiqueMakowski]

(see also #437)

Continuation of my thoughts on conceptualising priors and teaching prior specification in a useful way

The EasyPrior Framework

Priors exist in a three-dimensional space.

1. Fidelity

alternative names: coherence, alignment, appropriateness, ...

The most important dimension. This refers to whether a prior distribution is aligned with the target "true" distribution/range of the parameter.

• Well-matched: The distribution of the prior matches the target distribution. High-fidelity = Correct support and plausible shape. Examples:
  • A Beta prior distribution for a Parameter Beta-distributed parameter bounded within the 0-1 range
  • A Gaussian prior for a Gaussian-distributed parameter
• Tolerable: The distributions do not exactly match, but the support space is respected
  • A Cauchy prior for a Gaussian target distribution
  • A LogNormal prior for a Gamma distribution
• Inappropriate: Wrong support space
  • A Gaussian prior for a Beta-distributed (0-1 bounded) parameter
  • A Gaussian prior for a positive parameter

The first step of prior selection is to make sure the prior is not inappropriate, which requires some understanding of the parameters that we are estimating. However, thanks to the Central Limit Theorem, a Normal prior is in practice often a good choice.

2. Location

What is the most plausible value. Where do we put the most probability mass?

• Believer: the probability is on the expected effect. Traditional flavour of "Bayesian" philosophy, where we our prior beliefs match our expectations.
• Agnostic: Prior with the most mass at 0 (for unbounded parameters, must be symmetrical). The "practical" flavour of applied Bayesianism. Useful in NHST.
• Sceptic: Prior centred on the opposite of the expected effect. It can be used to test and assess the robustness of the findings.

3. Precision

How narrow/wide is our prior. How much influence does it have.

• Flat: no information from the prior. Note that a prior can be flat and inappropriate (e.g., a Flat prior for a 0-1 bounded parameter needs to be a Beta(1, 1) or a bounded Uniform. The support space still matters!)
• Wide: In-between
• Narrow

Example

Let's say we are trying to specify the prior for the SD parameter of participants random effect in a mixed model. This parameter is strictly positive, and let's assume its "true distribution" (a bit of a shaky concept) is Gamma(alpha=4, theta=4). This means the mean = 16, and mode = 12.

Fidelity	Location	Precision	Example Prior	Comment
Well-matched	Believer	Narrow	Gamma(40, 0.4)	High confidence around mean = 16
	Believer	Wide	Gamma(4, 4)	Matches true shape and location
	Agnostic	Narrow	HalfNormal(0, 5)	Peaks at 0, steep drop-off — agnostic but constraining
	Agnostic	Wide	HalfNormal(0, 20)	Less informative agnostic prior
	Sceptic	Narrow	Gamma(3, 1)	Peaks at 2 — sceptical of large SDs
	Sceptic	Wide	Gamma(2, 1)	Prior mass toward low SDs, but allows higher values
Tolerable	Believer	Narrow	LogNormal(2.5, 0.1)	Support matches, shape doesn't — very peaked near 12
	Believer	Wide	LogNormal(2.8, 0.4)	LogNormal matches range, looser belief near 16
	Agnostic	Narrow	Exponential(0.2)	Peaks at 0, drops off quickly — agnostic and regularising
	Agnostic	Wide	Exponential(0.05)	Very flat agnostic prior
	Sceptic	Narrow	LogNormal(0.1, 0.2)	Strong belief in small SDs (mode near 1)
	Sceptic	Wide	Exponential(1)	Skews mass toward small SDs
Inappropriate	Believer	Narrow	Normal(16, 1)	High belief near true value, but allows negative SDs
	Believer	Wide	Normal(16, 10)	Covers the range, but support includes negatives
	Agnostic	Narrow	Normal(0, 2)	Symmetric at 0 — violates support (negative SDs)
	Agnostic	Wide	Normal(0, 10)	Even wider, more mass on invalid values
	Sceptic	Narrow	Normal(-5, 1)	Strong belief in impossible (negative) values
	Sceptic	Wide	Normal(-5, 10)	Diffuse, but still fundamentally wrong support

We could do a 3D plot to illustrate this with various examples

The terminology and names might not be optimal.

[DominiqueMakowski]

good points, thanks :)

Fidelity

Yes indeed, the concept of "true" distribution is shaky. I think what I meant was true in a conceptual sense - "if we repeated the experiment an infinite amount of times on new samples" (hence invoking CLT) but it does feel weird to use frequentist concepts to justify picking an appropriate shape for the prior (though in practice, the most important bit IMO is coverage space).

Cool package indeed! @strengejacke could be worth mentioning in your prior paper?

Location

I meant that having priors centered at 0 does makes sense in particular in experiments trying to test whether an effect exists (a parameter != 0). In these cases, it's useful to have the possibility of shrinking the effect towards 0 (being more conservative), but not influence its direction.

differentiate between believer and sceptic, considering the expected effect is specified by the prior

Here I am making a difference between the prior as a statistical tool and the effect expected "externally" (from the literature, hypothesis and whatnot). In typical Bayesina philosophy these two should align, but I believe they can be dissociated: I think it is fine to "know" that some effect is likely to be 0.3, and have hypotheses on that, but use a prior of -0.3 (or 0). They are in a way counterfactual priors: "I make a statistical model assuming I believe something different from what I believe/expect"

Preicsion

Agreed. Although as mentioned in my first comment I find "strong/weak" prior as a bit ambiguous as it could also refer to the location (or at least to a combination of both: any prior specification that strongly influence the posterior). Hence I think it's clearer to avoid using strong/weak, and instead use precise/narrow vs. wide/flat, and "strongly directional" (? - better name welcome) for location

how much the posterior "looks like" the prior

But what if, say, there is infinitely OVERWHELMING evidence in the data that some parameter is Normal(3, 2). Both the location and uncertainty of that parameter are super strong (although you might argue that the SD should decrease as the evidence increases?).
In this case, priors wouldn't have much effect on the posterior: so a Normal(0, 100) would lead to the same posterior as a prior of Normal(3, 2). Although the prior == posterior, would you conclude that the prior is strong 🤔 ? Despite having no virtually no effect on the posterior?

That's something else I don't like about strong/weak, is that it implies (to me) "does it have an impact on the posterior", which is somewhat a post-hoc concept. Whereas terms like precise/flat and directional are somewhat orthogonal to the posterior (although precise/flat still arguably requires a minimum of domain knowledge of the parameter).

[strengejacke]

could be worth mentioning in your prior paper?
I also found this paper, but I thought this might be too far off for the discussion, as the paper is meant to be a rather "non-technical" introduction.
But I think the recommendations table could be replaced by your suggestion, or maybe combined or whatever.

[mattansb]

I am personally not a fan of trying to assign frequentists desiderata to Bayesian tools. So I still don't really follow what you mean by "true" posterior - even with repeated sampling. The marginal posterior across all samples? So like the "expected" posterior?

"I make a statistical model assuming I believe something different from what I believe/expect"

So like relaxing priors for estimation? I think this is usually done for one or both of the following reasons:

1. Protection against criticism of strong prior and the need to do prior robustness checks.
2. Being a-priori open to being wrong (in which case I would argue the formal belief is weaker than stated)

A wide prior centered on 0 will produce the same results as a wide prior centered on not-0. I really see location and precision as part of the same thing - how specific you want your priors to be. The is nothing inherently "default" to having a prior $N(0, 2.5\times SD)$.

But what if, say, there is infinitely OVERWHELMING evidence in the data that some parameter is Normal(3, 2).

I don't follow. If data is giving overwhelming evidence, it will be to more and more specific values, not stronger and stronger to the same range of values (I don't even know what the means). Same as MLE estimation - more data (=more evidence) smaller SEs and smaller CIs. So with strong evidence the posterior will be more narrow (more selective about which values are credible).

...so a Normal(0, 100) would lead to the same posterior as a prior of Normal(3, 2) [?]

Yes!

library(brms)

dta <- data.frame(
  y = rnorm(5000, mean = 3, sd = 2)
)

mod_congruent <- brm(
  y ~ 1,
  prior = prior(normal(3, 2), class = Intercept),
  family = gaussian(),
  
  data = dta
)

mod_diffused <- update(mod_congruent, prior = prior(normal(0, 100), class = Intercept))

mod_incongruent <- update(mod_congruent, prior = prior(normal(-3, 2), class = Intercept))


bayestestR::hdi(mod_congruent)
#> Highest Density Interval 
#> 
#> Parameter   |      95% HDI
#> --------------------------
#> (Intercept) | [2.95, 3.06]

bayestestR::hdi(mod_diffused)
#> Highest Density Interval 
#> 
#> Parameter   |      95% HDI
#> --------------------------
#> (Intercept) | [2.95, 3.06]

bayestestR::hdi(mod_incongruent)
#> Highest Density Interval 
#> 
#> Parameter   |      95% HDI
#> --------------------------
#> (Intercept) | [2.95, 3.06]

[DominiqueMakowski]

my last example was somewhat nonsensical, I need to think more

Specificity is an interesting term, and we could even reframe the framework with pseudo-psychometric accuracy/specificity terminology: your prior needs to be Accurate (i.e., with respect to the parameter-space (coverage) $\pm$ related to the distribution shape) and more or less Specific (combination of location & precision).

The is nothing inherently "default" to having a prior N(0, 2.5SD)

agreed

A wide prior centered on 0 will produce the same results as a wide prior centered on not-0

In practice, yes, but there is a philosophical difference in assigning the higher plausibility (relatively) on non-zero vs. zero

So I still don't really follow what you mean by "true" posterior - even with repeated sampling

I need to think more

[mattansb]

Here's another example (easy to make more with conjugate priors). Here we have:

$y \sim Binomial(p, N)$

The prior is: $p \sim Beta(3, 7)$ and the observed data is 3 successes out of 10.

We get the prior and the likelihood are in full agreement about the relative credibility of the values of p - but since this is additive, the posterior is more concentrated.