How Big Were Pandemic Learning Losses, Really?
The pandemic isn’t entirely behind us, not with hundreds of Americans dying each day at an annualized rate of more than 100,000. But the emergency phase of the country’s response is now far enough in the rearview mirror that retrospective assessments are beginning to feel a little less irresponsibly in medias res than they did even six months ago.
In the past few weeks, two big data sets have offered real insight into one of the pandemic’s more contentious subjects: the scale of learning loss experienced as a result of school closures.
In the first, the National Center for Education Statistics released its National Assessment of Educational Progress long-term trend data, drawn on its math and reading testing of thousands of 9-year-olds across the country. In the second, the Covid-19 School Data Hub, run by the lightning-rod economist Emily Oster, released more granular analysis of state-level testing data of children in grades three through eight, collected from 13 states.
What does the data say? At the highest level, the picture is a pretty complex and nuanced one. Measured by conventional testing, educational learning losses from disrupted school look real but sub-catastrophic, variable by geography, wealth status and other less quantifiable factors; and though they are not just short-term negative blips, they also appear to not be permanent, with students recovering at least some ground already.
But before I dig in further, I want to lay a few cards on the table, given how bitter and ideological the fights over school closures have been, so that what follows can be read in full transparency.
First: children and Covid. From almost the very beginning of the pandemic, I’ve written about the age skew of mortality risk — that the risk to the elderly was much higher than most people appreciated or than our policies really reflected, and that the risk to younger people, particularly young children, was much lower.
These patterns still hold, in an age of vaccination and boosters. A recent study of risk during the Omicron wave among the vaccinated and boosted showed that an 80-year-old was 30 times more likely to die from infection than a 50-year-old, an effect many times the size of even the largest comorbidities. But if I were writing those earlier pieces again today, I would emphasize much more the way that transmission, alongside infection fatality ratios, determine overall risk to an individual. That phenomenon helps explain why, though the proportion of children infected who die from Covid hasn’t grown, the number of pediatric deaths in the U.S. since the fall of 2021 and the onset of the Omicron waves is more than twice as high as in the whole 18 months of pandemic spread that preceded it.
Second: school closures. Because of the relatively low risk to children themselves, I would have preferred that “last to close, first to open” had been the guiding principle of school policy from the very beginning and had been broadly applied by the fall of 2020. The pandemic offers countless opportunities for retrospective counterfactuals, but I thought at the time and still do now that it was unfortunate and damaging that so few schools seemed able to prepare over the spring and summer to open safely by that September — embracing rapid or pooled testing for screening, utilizing outdoor space in parts of the country where it was possible and using new ventilation and air-filtration systems where it was not.
But I also think that it was not unreasonable amid an enormous amount of broader anxiety, precaution and disruption for schools to close in the spring of 2020. “Last to close, first to open” doesn’t mean “never close.” And though the age skew was already clear, back then the country was much less focused on assessments of individual risk and much more focused on reducing transmission, on the principle that continuing chains of infection would ultimately lead to more vulnerable people.
And third: testing. I’m somewhat skeptical about how much test scores reveal, particularly at the individual level, having come of age at an absolutely remarkable elementary school, Central Park East, founded among others by my mother, where testing was a dirty word and teachers often wore T-shirts printed with “High stakes are for tomatoes.” But I also believe that while school closures did a lot more than just affect test results, at the aggregate level, testing does tell us some things.
First, over the first year of the pandemic, American schoolchildren experienced an obvious and unfortunate setback in learning and achievement. In all but two of the 13 states examined by Oster’s Covid-19 School Data Hub, proficiency rates fell by between 5 and 11.5 percentage points. (In Virginia, math declines in particular were much larger.) In some places, according to that analysis, the drops were from relatively high standards of achievement: in one large cohort of Ohio schools, for instance, from 73 percent proficiency in math to 61 percent. In others, distressingly, they were from much lower levels: Across much of Georgia, reading proficiency before the pandemic had been below 40 percent. All told, the impact is significant at the state level: Many thousands of children fell behind where they would have been if they had an uninterrupted experience of schools.
Second, the setbacks may not be all that big, all things considered. In many of these states, despite significant disruption, declines were relatively modest — a five or ten point drop less like school systems suddenly abandoning half of their students than like states that had been grading out at, say, a “C+” level now grading out at a “C-” level.
When The Times first reported on the testing data, the headline was that “the pandemic erased two decades of progress in math and reading.” But a closer look at that second set of data gives a subtler picture. The math and reading tests are scored out of a maximum of 500. In 2022, the average math score was 234, down from an average of 241 in 2020, just before the pandemic, and from a peak of 244 in 2012. (You can find the results here.) In 1999, the average score was 232, and all of the scores over the past two decades fall in that range, between 232 and 244. The seven-point drop in average score from 2020 to 2022 was only about twice as large as the three-point drop observed in 2020 from the average score on the previous assessment.
For reading, the average score in 2022 was 215, down from 220 in 2020 and from a peak of 221 in 2012. The average score had been 220 in 2008 and 216 in 2004 — meaning that for a couple of decades now, including the pandemic period, average scores have been within a relatively narrow range. The drop between 2020 and 2022 was of roughly the same scale (five points) as the jump between 2004 and 2008 (four points).
These figures are just averages, of course, and do show real declines. But they do not imply that third graders are now at the level of kindergartners, or anything like that. Instead, the data suggests that schools performed in the midst of pandemic disruption as well as schools did around the year 2000, for math, and around the year 2004, for reading.
This data is partial, of course, and more will be forthcoming to fill in our understanding. Perhaps the picture will grow darker. But to trust that the NAEP long-term trend data is representative means understanding that these, in the end, were the terms of the bargain haphazardly struck in the spring 2020: that in exchange for an imperfect, overlong and disruptive public-health response, students would, for a period of time, not reach the achievement levels of American schoolchildren just before the pandemic but the achievement levels they might’ve had a decade or two earlier.
If this outcome had been known with perfect clarity back in the spring of 2020, what would the public response have been? Surely some would have objected to closures, including those who didn’t see the pandemic as much of a threat. But overall I don’t think that parents, teachers and administrators or political leaders would have been horrified at the trade-off. Given everything else being done at the time to reduce spread — citywide lockdowns, months of social isolation and canceled events and gatherings — I think that cost would have been widely seen as pretty manageable. Presumably many of these measures could have been managed more effectively, and perhaps applied in more targeted ways. But given how long the school disruptions lasted and how significant they often were, the declines seem impressively small. (And since teachers unions have gotten a lot of criticism for delaying reopenings — even though surveyed parents were also often reluctant to return to in-person school — it may be worth mentioning that the test scores suggest that given the circumstances, the country’s teachers managed relatively well, too.)
Third, the size of the effect varied considerably. In Colorado, where only 22 percent of students had a lot of in-person schooling, test scores declined by only 5.4 percentage points between 2019 and 2021. In Minnesota, where 24 percent did, the cost was higher: 10.6 percentage points. At the state level, it’s hard to even see clearly the relationship between school closures and pandemic learning loss, and when Oster breaks down the data into cohorts by levels of in-person schooling, the picture doesn’t grow all that much clearer. In Indiana, for instance, scores in “very low” levels of in-person instruction dropped by 10 percentage points in reading and about three in math, while those with “very high” levels of in-person school fell by about eight points in each. In Minnesota, schools with “low” in-person levels lost about seven points in reading and nearly 14 in math, while schools with “high” levels lost nine points and 13 points.
That said, several earlier and more granular studies have shown stronger correlations between remote learning and testing declines at the district level, which suggests that aggregating district data probably folds a whole lot of confounding variables into what was already a complicated-to-unpack equation — and that remote learning, though not the only factor, did have a significant and meaningful negative effect. Poverty certainly plays a significant role, too, exacerbating learning losses, as previous studies demonstrating the much higher costs of school closures in poor districts have also found. A similar dynamic applies with race, and while high-performing students suffered relatively small setbacks, low-performing students registered larger losses. Closer examinations yield an even more complicated picture: In a useful meditation on the testing data for KQED, Jill Barshay points out that reading scores were relatively stable through the pandemic, in urban school districts and across the West, while math declines were much more universal.
Fourth, we are already seeing a rebound effect, reflecting the fact that across the country, students appear to be learning at “normal” pace again. But the rebound effect is not as robust as one might have hoped, with some experts suggesting it will take three to five years to recover from the setbacks. Overall, Oster estimates, students have already made up about 30 percent of their pandemic losses, as measured by the 2022 tests, though again there is a range: Colorado recovered more than half of its losses, performing in 2022 only 2.6 percentage points behind its prepandemic baseline, whereas in Indiana, students recovered only about 10 percent of their losses, and this year were almost eight points behind. Which is to say, we don’t yet know how long-lasting these setbacks will be, but we know that they are shrinking. And perhaps there is something to learn from Mississippi in particular, where, according to the 2022 test results, students have already regained as much as 80 percent of their pandemic learning loss.
How does all this add up? Not all that neatly — which may be the most important point. Throughout the pandemic, we have conjured narratives out of messy experience, trying to extract obvious lessons or apportion responsibility when reality is more complicated and contradictory than is emotionally or ideologically palatable. In the summer of 2020, we railed against people running outside or going to lake parties, and then against those traveling home for the holidays, PCR result in hand. We talked about “Red Covid” — as it supposedly became a disproportionately Republican illness — when vaccination rates were never much higher and death rates never much lower among the poor, poorly educated and racially disenfranchised than among Republicans. Because we knew that vaccines significantly reduced individual risk, we used the phrase “vaxed and done” even as more vaccinated than unvaccinated Americans were dying. We worried about teenage depression when eating disorders may have been the biggest mental-health impact of the pandemic for adolescents.
And when it came to schools, we fought over the consequences of remote learning as though the present health and future well-being of an entire generation of students was on the line. What we are starting to see more clearly now is that those consequences, though worth reflecting on, were probably not as large as either side feared. “Students haven’t regressed,” Barshay emphasizes, comparing the dynamic to a cross-country road trip. “Imagine that students were traveling at 55 miles an hour, ran out of gas and started walking instead,” she says. According to testing data, “now they’re back in their cars and humming along again at 55 miles an hour. Some are traveling at 60 miles an hour, catching up slightly, but they’re still far away from the destination that they would have reached if they hadn’t run out of gas.”
Next time, we should try to do better. But all things considered, the best data we have at the moment suggest that school closures were probably a fog-of-war-style social and political stumble, not, as some have suggested, a “disastrous, invasion-of-Iraq magnitude (or perhaps greater) policy decision.”
David Wallace-Wells (@dwallacewells), a writer for Opinion and a columnist for The New York Times Magazine, is the author of “The Uninhabitable Earth.”
