Irreproducibility
I can sometimes find it difficult to even wrap my mouth around the word, but according to vocabulary.com, 'irreproducibility' refers to 'the trait of not being dependable (n)or reliable.' In the case of research, it has come up for social sciences and also economics. If you haven't heard about it, one of my favourite podcasts Planet Money gives a great overview of the issue. You can also find a brief summary of the issues in Nature, and full details of the research in Science, and follow up research to propose solutions to the problem in Nature.
The Problem
- Psychologist Brian Nosek initiated a project to examine the reproducibility of social psychology studies
- He gathered a team of researchers who attempted to reproduce the results of 100 studies published in reputable journals
- When the results were gathered, only 39 of the studies were considered to be reproduced
While there is some variability in what was considered to be reproduced (as can be seen in the image to the right from one of the Nature articles), these were still concerning findings and threatened the confidence placed in the scientific process for published research.
The Causes
Some of the potential causes for these issues, as suggested by Nosek and others, include:
- Major research journals do not typically publish studies that produce non-significant results
- Currently close to 0% of available research funding is directed towards reproducing the results of other studies
- Researchers can inadvertently change sample sizes or other seemingly innocent/appropriate modifications if they appear close to a significant result
The Solutions
Some of the solutions proposed by Nosek and others include:
- Dedicating 3% of research funding to reproducing studies
- Requiring researchers to indicate their intention to complete a study, with planned analysis methodologies, in an accessible repository
- Using this repository to confirm that researchers maintained their proposed methodology
- This repository will then also provide a collection of non-significant results for comparison with future research
The Implications
But what does any of this have to do with the field of data science? At least to me, there are some quite clear connections.
Essentially what Nosek et al. demonstrated is that current practices associated with reputable, peer-reviewed journals have appeared to increase the likelihood of Type I errors in published research.
Given that the hypothesis testing that is involved in data analysis is often an iterative process, based on the examination of data, and allowing the data to inform the questions asked, and review of these methodologies likely does not face the same scrutiny as those of peer-reviewed journals, it would seem that data science testing and modelling can be susceptible to the same issues.
The question is, how is this being addressed? My truthful answer at this time is, I don't know!
It is something that I am looking forward to asking more informed people than I. Maybe I will be back in the future with edits if I get some answers. :)