Economics Has a Replication Problem
"Economics research is usually not replicable."
That's the conclusion of economists Andrew C. Chang and Phillip Li in a new study released as part of the Finance and Economics Discussion Series at the Federal Reserve.
Analyzing research from thirteen top economics journals, Chang and Li were able to replicate the findings of just 29 of the 59 papers they scrutinized, and that was with the assistance of the original authors. The duo was also very generous with their definition of "successful replication." As long as they received a result that pointed in the same direction as the original, the original was considered replicated.
"For example, if the paper estimates a fiscal multiplier for GDP of 2.0, then any multiplier greater than 1.0 would produce the same qualitative result," they explained. "We define success using this extremely loose definition to get an upper bound on what the replication success rate could potentially be."
Believe it or not, a near 50% replication rate is actually an improvement for economics. Back in 2006, researchers examined more than 150 economics articles and found that only fifteen of those could be replicated. The researchers were damning in their assessment:
Distinct from most sciences, economics has not fully embraced the scientific method; in particular, there is no tradition of replication in economics. Results published in economics journals are accepted at face value and rarely subjected to the independent verification that is the cornerstone of the scientific method. Most results published in economics journals cannot be subjected to verification, even in principle, because authors typically are not required to make their data and code available for verification. Moreover, when such a requirement does exist at a particular journal, it can be (and is) ignored with impunity.
Economics has long been lambasted as unscientific, and while Chang and Li's study shows some progress within the field, there is still much work to be done. The duo noted that most of the issues with replication arose because the authors didn't actually provide the means to reproduce their studies. Indeed, most economic research is published without presenting the original data or code used to produce the results. Without either of those vital pieces of information, the finding is simply taken at face value.
But science doesn't work that way. We know something to be true only when it is harshly scrutinized and repeatedly reinforced with solid evidence. If economists want their findings to be taken seriously, they need to start holding themselves and their research to higher standards.
Source: Chang, Andrew C., and Phillip Li (2015). “Is Economics Research Replicable? Sixty Pub- lished Papers from Thirteen Journals Say ”Usually Not”,” Finance and Economics Dis- cussion Series 2015-083. Washington: Board of Governors of the Federal Reserve System, http://dx.doi.org/10.17016/FEDS.2015.083