Math in a Time of Excel: Economists' Error Undermines Influential Paper
Now the most influential product of their collaboration – an argument widely considered to have helped lay the basis for the West's recent shift toward austerity economics – has been shown to rest in large part on a simple error, a spreadsheet coding mistake discovered by a 28 year-old graduate student at the University of Massachusetts Amherst named Thomas Herndon.
"I almost didn't believe my eyes when I saw the basic spreadsheet error," Herndon told Reuters. "I was like, am I just looking at this wrong? There has to be some other explanation." Herndon asked his girlfriend for confirmation that he wasn't missing something. He was right, she said.
The story begins with a large-scale research project carried out by Reinhart and Rogoff, investigating hundreds of years of financial crises around the world. The result was a well-received 2009 book, This Time Is Different: Eight Centuries of Financial Folly. They also published a paper based on a selection of their data, titled "Growth in a Time of Debt" (2010), which claimed to show that economic growth slows appreciably once a country's public debt to GDP ratio gets above 90 percent.
It was an argument bound to receive attention from the press and policymakers, because, according to some projections, the U.S. was set to pass that purported 90 percent red line in the coming decade. And the Reinhart-Rogoff result achieved notoriety on both sides of the Atlantic: Reinhart testified before the National Commission on Fiscal Responsibility and Reform; Paul Ryan cited the study in his 2013 budget, "The Path to Prosperity: A Blueprint for American Renewal"; and European Union Economic and Monetary Affairs Commissioner Olli Rehn spoke in February of "serious academic research" that indicated an economic danger zone above 90 percent debt-to-GDP. (The G20 countries have said they will soon consider a proposal "to cut public debt over the longer term to well below 90 percent of gross domestic product," The Fiscal Times reports.)
The Washington Post editorial board even presented the "Growth in a Time of Debt" finding as established economic fact. Writing about the risks of supposedly overly rosy economic growth and spending projections, the paper cautioned, "debt-to-GDP could keep rising – and stick dangerously near the 90 percent mark that economists regard as a threat to sustainable economic growth."
Needless to say, this being an academic question, there was disagreement all along. Several economists argued for "reverse causation," suggesting that high levels of debt are in fact caused by slow economic growth, rather than conversely. Others questioned the data that Reinhart and Rogoff chose to focus on, saying the authors excluded evidence that didn't match their take-home result.
But the fatal blow came when Herndon attempted to replicate the eminent professors' findings for a class assignment. He thought he would arrive at the same results but offer a different explanatory conclusion; instead he found that the numbers weren't right. When Reinhart and Rogoff provided their data spreadsheet, Herndon saw why: In the words of a paper he wrote with his teachers, "a coding error in the RR working spreadsheet entirely excludes five countries, Australia, Austria, Belgium, Canada, and Denmark, from the analysis." This spreadsheet error "is responsible for a -0.3 percentage-point error in RR's published average real GDP growth in the highest public debt/GDP category."
So that slightly negative average growth rate for countries with debt-to-GDP ratios was illusory, the product of an Excel error; 2.2 percent is the actual GDP growth rate for countries above the 90 percent mark.
Herndon and his co-authors found other errors, as well; Mike Konczal summarizes their findings at Next New Deal. Confronted with Herndon et. al's evidence, Reinhart and Rogoff released a response which essentially said "this changes nothing." They later conceded the coding mistake, saying, "It is sobering that such an error slipped into one of our papers. We do not, however, believe this regrettable slip affects in any significant way the central message of the paper." They don't accept the other criticisms, which concern data manipulation (selective exclusions and unconventional weighting).
Herndon's teacher and co-author Michael Ash told Bloomberg that the UMass-Amherst team's new calculations do show "a modest diminishment of growth" in high-debt countries, but not "the stagnation or decline" claimed by Reinhart-Rogoff. The point, in Konczal's summation, is that "there's no magic number out there":
The debt needs to be thought of as a response to the contingent circumstances we find ourselves in, with mass unemployment, a Federal Reserve desperately trying to gain traction at the zero lower bound, and a gap between what we could be producing and what we are. The past guides us, but so far it has failed to provide evidence of an emergency threshold. In fact, it tells us that a larger deficit right now would help us greatly.
But an identifiable threshold was apparently alluring to the Harvard academics, and advantageous to political elites. The tide of opinion among policymakers may be shifting, however: Olli Rehn told Reuters that the euro zone will pivot back towards attempting to promote growth, rather than cut budget deficits. Whether this decision was influenced by the Reinhart-Rogoff affair was not discussed.