Missing robustness analysis code and plots in two-sample ALE example documentation

[psylym]

Summary

The example script for the two-sample ALE meta-analysis (CBMA subtraction & conjunction analysis) does not appear to include any code or plots demonstrating the robustness analysis (e.g., visualization of robust activation areas), even though such plots are shown in the blog article referenced.

Additional details

• NiMARE version:v0.5.1

What were you trying to do?

I attempted to replicate the robustness analysis plot shown in the blog-cited paper (Alexander Enge, Rasha Abdel Rahman, and Michael A Skeide. A meta-analysis of fMRI studies of semantic cognition in children. NeuroImage, 241:118436, 2021. URL: https://doi.org/10.1016/j.neuroimage.2021.118436, doi:10.1016/j.neuroimage.2021.118436), using coordinate-based data and the example subtraction analysis script from the official NiMARE documentation.

What did you expect to happen?

I expected the example script to include or reference the code that generated the robustness visualization — i.e., the plot showing brain regions with robustness above a specific threshold, similar to the figure shown in the paper.

What actually happened?

The script executed successfully for the two-sample ALE and subtraction analyses, but it did not include any robustness evaluation or plotting code. The robustness results/visualizations shown in the blog post are not reproducible based on the current example script.

Reproducing the bug

[jdkent]

Hi @psylym,

The example could include the figures, thanks for bringing that to our attention. In the meantime, I have a couple resources for you to use as a reference to replicate those plots/perform a robustness analysis.

Here is the code used in the published article, specifically, here is the jackknife analysis.

Additionally, we have our own jackknife (leave-one-out) analysis implemented in NiMARE: https://nimare.readthedocs.io/en/stable/generated/nimare.diagnostics.Jackknife.html

The output from NiMARE's Jackknife places emphasis on the relative contribution of each study on the resulting clusters, by taking the statistical values generated by leaving the experiment out versus having all the experiments together.

For example, suppose the average z value for a cluster was 3.85. In that case, if we left out one experiment and the value was still 3.85, the relative contribution of that experiment is calculated as $1 - (3.85 / 3.85) = 0$, indicating the experiment had 0 contribution to that cluster. If leaving out the experiment resulted in a new statistical value of 0, the relative contribution of the experiment is $1 - (0 / 3.85) = 1$, suggesting that experiment was the sole contributor to that cluster. For the result to be considered robust, the ideal would be for each experiment to contribute equally to the cluster, so if there were 8 experiments each experiment should contribute .125 to the cluster.

Let me know if you have additional questions!
James

[psylym]

Thanks a lot for your attention! Really appreciate your help!