The Replication Crisis
Last week I (semi-publicly) asked a question whose implications are terrifying.
I was doing a live Q&A session at Liberty Classroom with Professor Jeffrey Herbener of Grove City College. I asked him something like this:
Jeff, if I were to ask you about the state of the economics profession, you would surely tell me that it's an absolute wreck, dominated by people employing the wrong method, who are more devoted to modeling than to the real world, and who are going down intellectual dead ends.
And that would be the good news. The bad news would be that economics is being willfully perverted in order to provide intellectual cover for the cronyism and thieveries of political regimes around the world.
Now what are the chances that economics is the only screwed up one in academia, that you just happen to belong to the only corrupt one? Could the rot go deeper? What other disciplines are producing more nonsense than sense?
Jeff's answer involved something I had somehow been unaware of, but that I think you, dear reader, will want to know about if you do not already.
He spoke about something called the "replication crisis," which has swept through the field of psychology over the past decade.
Researchers began finding that in case after case after case, they could not replicate the results that had been reported in peer-reviewed studies, even when carrying out the studies in exactly the same manner.
The studies, in other words, were fraudulent.
A variety of explanations have been proposed for the problem -- poor study design, a bias toward interesting or unusual results, or a bias toward studies that find positive results rather than those that show no effect.
Medical research has had a similar problem. "There is a worrying amount of fraud in medical research," ran a headline in The Economist in 2021.
Richard Smith, former editor of the British Medical Journal (better known as the BMJ), insisted that same year that the problem of research fraud wasn't one of "bad apples" but rather of "bad barrels" or indeed of "rotten forests or orchards."
He went on:
Stephen Lock, my predecessor as editor of The BMJ, became worried about research fraud in the 1980s, but people thought his concerns eccentric. Research authorities insisted that fraud was rare, didn’t matter because science was self-correcting, and that no patients had suffered because of scientific fraud.
All those reasons for not taking research fraud seriously have proved to be false, and, 40 years on from Lock’s concerns, we are realising that the problem is huge, the system encourages fraud, and we have no adequate way to respond. It may be time to move from assuming that research has been honestly conducted and reported to assuming it to be untrustworthy until there is some evidence to the contrary.
The rot is everywhere.
I was doing a live Q&A session at Liberty Classroom with Professor Jeffrey Herbener of Grove City College. I asked him something like this:
Jeff, if I were to ask you about the state of the economics profession, you would surely tell me that it's an absolute wreck, dominated by people employing the wrong method, who are more devoted to modeling than to the real world, and who are going down intellectual dead ends.
And that would be the good news. The bad news would be that economics is being willfully perverted in order to provide intellectual cover for the cronyism and thieveries of political regimes around the world.
Now what are the chances that economics is the only screwed up one in academia, that you just happen to belong to the only corrupt one? Could the rot go deeper? What other disciplines are producing more nonsense than sense?
Jeff's answer involved something I had somehow been unaware of, but that I think you, dear reader, will want to know about if you do not already.
He spoke about something called the "replication crisis," which has swept through the field of psychology over the past decade.
Researchers began finding that in case after case after case, they could not replicate the results that had been reported in peer-reviewed studies, even when carrying out the studies in exactly the same manner.
The studies, in other words, were fraudulent.
A variety of explanations have been proposed for the problem -- poor study design, a bias toward interesting or unusual results, or a bias toward studies that find positive results rather than those that show no effect.
Medical research has had a similar problem. "There is a worrying amount of fraud in medical research," ran a headline in The Economist in 2021.
Richard Smith, former editor of the British Medical Journal (better known as the BMJ), insisted that same year that the problem of research fraud wasn't one of "bad apples" but rather of "bad barrels" or indeed of "rotten forests or orchards."
He went on:
Stephen Lock, my predecessor as editor of The BMJ, became worried about research fraud in the 1980s, but people thought his concerns eccentric. Research authorities insisted that fraud was rare, didn’t matter because science was self-correcting, and that no patients had suffered because of scientific fraud.
All those reasons for not taking research fraud seriously have proved to be false, and, 40 years on from Lock’s concerns, we are realising that the problem is huge, the system encourages fraud, and we have no adequate way to respond. It may be time to move from assuming that research has been honestly conducted and reported to assuming it to be untrustworthy until there is some evidence to the contrary.
The rot is everywhere.
