Ever wondered what would happen if 70 different groups analyzed the same fMRI dataset? The pre-print of our project is out ! 1/9
Variability in the analysis of a single neuroimaging dataset by many teams #biorxiv_neursci
Glad that people did this study, but the results are kinda horrifying. 70 fMRI teams analyzed the SAME DATA for 9 pre-determined hypotheses. At best, 85% of teams agreed one hypothesis was significant, and 94% agreed that three more were not.
All analyses in this paper are 100% reproducible (thanks to ) with open data; see for details on how to run it yourself. Let us know what you think and add your comment in biorXiv 9/9
Variability in the analysis of a single neuroimaging dataset by many teams "...our findings highlight the fact that it is hard to estimate the reproducibility of single studies that are performed using one single analysis pipeline"
70 teams analyzed 1 #neuroimaging dataset to answer 9 questions; each team ended up with a unique workflow to analyze the data, highlight considerable degrees of freedom.
Thanks so much for your contribution and creating this thread and adding the handles - WOW we'll recruit you also to the mgmt team next time :) BTW: finally have all authors listed!
Study at Data are public too #opendata
70 teams analyzed the same functional magnetic resonance imaging (fMRI) data, testing 9 hypotheses. No 2 teams chose identical analysis plans & results varied considerably, but meta-analysis showed some convergence. Prediction markets were too optimistic
fascinating >> Variability in the analysis of a single neuroimaging dataset by many teams
IMO, illustrates the need for theory-driven best practice analytic frameworks🛑Variability in the analysis of a single neuroimaging dataset by many teams🛑
#genemappers19 SM: Heterogeneity between scanners and software. Needs lots of care to harmonise and pool imaging data. ENIGMA publish protocols for each paper. No gold standard pipeline, big differences in outcomes - see for example.