Andrew C. Chang and Phillip Li undertook a study of "67 papers published in 13 well-regarded economics journals" for the US Federal Reserve and attempted to replicate their conclusions, using the code and data-sets provided by the authors: in two-thirds of cases, they were unable to replicate the findings without help with the authors; with the help of authors, they were still only able to replicate 49% of the papers.
In this article, we attempt to replicate 67 papers from 13 well-regarded economics journals
using author-provided data and code replication files. Improving on existing work evaluating
the state of replication in economics, our sampling frame is broader across different journals
and covers a large number of original research articles. We replicate 22 of 67 papers (33%)
by using only the authors’ data and code files, and an additional 7 papers (for a total of
29 papers, 43%) with assistance from the authors. The most common cause of our inability
to replicate findings is that authors do not provide files to the journal replication archives,
which constitutes approximately half of our failed replication attempts (21 of 38 papers,
55%). Because we are able to replicate less than half of the papers in our sample, we
conclude that economics research is generally not replicable.
Is Economics Research Replicable? Sixty Published Papers from
Thirteen Journals Say ”Usually Not” [Andrew C. Chang and Phillip Li/Federal Reserve]
