Ever wondered what would happen if 70 different groups analyzed the same fMRI dataset? The pre-print of our project is out ! 1/9
Variability in the analysis of a single neuroimaging dataset by many teams #biorxiv_neursci
6 Take home: share uncorrected data. Be aware of analytic flexibility in every scientific field with high-dim data and complex analytic choices. We’re working on multiverse tools for individual studies . Read here #OHBMx
Glad that people did this study, but the results are kinda horrifying. 70 fMRI teams analyzed the SAME DATA for 9 pre-determined hypotheses. At best, 85% of teams agreed one hypothesis was significant, and 94% agreed that three more were not.
All analyses in this paper are 100% reproducible (thanks to ) with open data; see for details on how to run it yourself. Let us know what you think and add your comment in biorXiv 9/9
If you can't pierce the paywall, here is the preprint
If there are less than a thousand data points, all but the simple statistical question can and will be gamed (sometimes unconsciously), partly for lack of model selection. An example in neuroimaging I no longer trust such endeavors, including mines. 5/8
Variability in the analysis of a single neuroimaging dataset by many teams "...our findings highlight the fact that it is hard to estimate the reproducibility of single studies that are performed using one single analysis pipeline"
70 teams analyzed 1 #neuroimaging dataset to answer 9 questions; each team ended up with a unique workflow to analyze the data, highlight considerable degrees of freedom.
Thanks so much for your contribution and creating this thread and adding the handles - WOW we'll recruit you also to the mgmt team next time :) BTW: finally have all authors listed!
Those guidelines were used to curate the reports of the 70 teams of the Neuroimaging Analysis Replication and Prediction Study (NARPS) - a one data set many analysts project for fMRI website: preprint:
The one I took part in has a comparison of scientists who took part versus those who didn't. Not exactly what you asked for though.
70 teams analyzed the same functional magnetic resonance imaging (fMRI) data, testing 9 hypotheses. No 2 teams chose identical analysis plans & results varied considerably, but meta-analysis showed some convergence. Prediction markets were too optimistic
IMO, illustrates the need for theory-driven best practice analytic frameworks🛑Variability in the analysis of a single neuroimaging dataset by many teams🛑
#genemappers19 SM: Heterogeneity between scanners and software. Needs lots of care to harmonise and pool imaging data. ENIGMA publish protocols for each paper. No gold standard pipeline, big differences in outcomes - see for example.
fascinating >> Variability in the analysis of a single neuroimaging dataset by many teams