The Polling Systemically Overestimated Democratic Results At Unprecedented Levels
The polling in the 2020 election had systematic problems that caused them to under-measure support for Republicans and Trump to a greater extent than any past election in more than a decade (including the 2016 election which was the runner up), and to over estimate support in favor of Democrats and Biden, in particular
This is something that is called technically "bias" in a poll (i.e. a lean in one direction due to errors across the board) as opposed to mere "error" which is disparity from the predicted outcome which does not itself lean one way or the other in any particular case, but isn't terribly accurate. In relative terms, the polling was largely on target, but the polling results were consistently more Democratic leaning than the actual outcomes. (I do not suggest that any pollster tried intentionally to get inaccurate results that favored Democrats and believe that they sincerely did their best based upon all evidence available to me. And, polling averages from disparate pollsters with different agendas would tend to filter out that kind of issue.)
It cannot plausibly be explained as simply being within the normal margin of error (the citation is to my own blog and is restated in part below; the link is included because the link provides access to more of specific factual results).
The 538 post-election analysis in 2020 looks a lot at historical "error" rates but contains some critical flaws.
First, a lot of the historical data compares a single poll (in the Gallup era) or a single polling average (national polling) to the result, and not a full set of state by state polling averages.
Second, it ignores the elephant in the room, which is that in 2020 the vast majority of significant polling errors overestimated Democratic performance. Polling error looks at how much the polls differed from the actual result. If half the races overestimate Biden performance by 4 points and half the races overestimate Trump performance by 4 points, the average polling error is 4 points, but the average lean is 0 points. But, if all of the races overestimate Biden performance by 4 points, the average polling error is still 4 points, but the average lean is also 4 points, which is a much bigger concern. The average error in the 2020 polling was high but not unprecedented. But the magnitude of the partisan lean in those errors in 2020 was unprecedented for a wide basket of separate polling averages.
I took the polling averages in all 50 states and the District of Columbia in the Presidential race and in each Senate race, complied by the Five Thirty Eight blog and compared them to the actual election results myself.
In the Presidential race (50 state polls, D.C. and the national polling average) the average pro-Democrat lean was 4.42 percentage points (compared to a previous record of 3.2 percentage points in 2016 and a 2.6 percentage point record lean pre-Trump and post-1998).
The pro-Democrat lean in the U.S. Senate polling in 35 races in 2020 was an average of 6.99 percentage points (compared to a previous record since 1998 of 4.8 percentage points in 1998).
(The mean error was somewhat larger than the mean Democratic lean. But adjusted for the Democratic lean in 2020 polling, as determined after the fact, the residual error in 2020 was smaller than in 2016, which is what was expected before the election since historical factors that favor higher error rates were absent.)
Keep in mind that each of these data points compares the actual results to a polling average of many polls by different polling operations in each race, not to a particular poll by one pollster or even a series of polls by one pollster. The averages were compiled from hundreds of individual polls, all recent and screened to keep out junk polling operations. The averaging method naturally makes the average more robust by screening for "house bias" to a great extent, reducing the expected margin of error relative to any individual poll, and screening out much the potential random sampling error. Essentially all of the lean towards one party or another has to be explained by systemic polling industry-wide issues.
Of 51 state polls in the Presidential race, the national Presidential rate polling and polling in 35 U.S. Senate races, a total of 87 polling averages, 79 (91%) erred in favor of Democrats.
Of the 87 polling averages, 54 (63%) were off by more than 4 percentage points (easily more than one standard deviation for a weighted average of election polls), with 98% of these large errors in favor of Democrats and the one large error in favor of a Republican coming in a Senate race without a Democrat running.
If the polls were not biased, you would expect a mean error in favor of Democrats of close to zero for 87 polling averages and mean errors overall of about half what we saw, with about equal numbers of polls biased in favor of Democrats and of Republicans both in terms of any errors, and in terms of large errors (about 43-44 errors for each party), and 28 or fewer large errors (about 14 errors for each party).
This level of polling bias is bigger than any election cycle polling bias in the 1998-2018 time period and about 33% greater than the next runner up in the Presidential races and 100% worse in the Senate races, than the next runner up, which was the 2016 Presidential election, which was biased in the same direction, for what in hindsight looks like the same reasons.
(538 source in 2018 article)
A lean like the one shown in these charts (which the lean in 2020 being record high) comes from only one source: non-random, systemic sampling errors. In 2016 and 2020 that non-random sampling error strongly appears to have been Trump-centric (although it might persist beyond him if attitudes towards surveys remain partisan). In other years, it was less severe and had different sources.
This is particularly notable because a priori the expectation was that polling error would be lower than in 2016 and would less strongly lean towards Democrats.
Years with high polling error are frequently associated with either lots of undecided voters (the New York Times flagged this as a key factor in polling errors in 2016) and/or a strong third-party performance. But in 2020, there were far fewer undecided voters than in 2016, and third-parties were not performing as well leading up to the election (or in the electoral outcomes themselves) as in 2016.
Pollsters across the board made efforts to increase cell phone coverage in their polling and to adjust the raw data towards demographics whose turnout had been underestimated in 2016. See also here.
High turnout also usually favors Democrats and high turnout of Democratic early and mail in voters in 2020 relative to any prior year (in part due to COVID-19) suggested that turnout models used to adjust raw polling results might underestimate Democratic candidate performance.
This isn't just my analysis. For example, an interview with Nick Beauchamp, assistant professor of political science at Northeastern likewise discounted the notion that it was just normal error:
There are a number of possible explanations, Beauchamp says. One is
that specific polls undercounted the extent to which certain
demographics—such as Hispanic voters in specific states—shifted toward
President Trump.
Another is that, just as in 2016, polls undercounted hard-to-reach
voters who tend to be less educated and more conservative. Beauchamp
is less convinced that “shy” Trump voters deliberately misrepresented
their intentions to pollsters.
“Whatever the cause, it has huge implications not just for polling,
but for the outcome of the presidential and Senate elections,”
Beauchamp says. “If the polls have been this wrong for months, since
they have been fairly stable for months, that means that campaign
resources may have also been misallocated.” . . .
This year’s polling errors, Beauchamp said, were “enormous” even
compared to 2016, when polls failed to predict Trump’s defeat of
Democratic nominee Hillary Clinton.
Indeed, just days before the presidential contest, FiveThirtyEight
founder Nate Silver predicted that Biden was slightly favored to win
Florida and its 29 Electoral College votes. The race was called on
election night with Trump comfortably ahead. . . .
“I think Nate Silver and the other pollsters are saying ‘Well that’s
just within the bounds of systematic polling,’ but it seems awfully
large to me for just that,” Beauchamp says.
The Possibility Of Unexpectedly High GOP Turnout Isn't An Explanation That Can Explain This Systemic Error
If the polling error was due to more post-polling decisions to go to the polls made by Trump supporters than Biden supporters, this would show up in levels of partisan turnout. Turnout was expected to be high in advance of the election. And, while turnout was higher in the 2020 election as a percentage of the eligible voter population (VEP) since the year 1900, turnout was high for Democrats as well as for Republicans (whose turnout varies much less from year to year than Democratic turnout does). There just isn't good evidence of this kind of turnout disparity large enough to account for the over performance of both Trump and Republican U.S. Senate candidate, relative to the polling. Notably, GOP turnout was not record high in many of the swing states where polling was badly off. For example, yes, white Evangelical Christian voters turned out in phenomenal numbers. But this happens every year and was expected.
Put another way, the demographic balancing done in the polls in 2020 based upon the shortcomings of the 2016 polling basically got it right. The predicted demographic mix of voters on election day and the actual demographic mix of voters on election day weren't seriously and systemically off. Instead, what appears to have happened is that almost every demographic polled had more Trump supporters on election day than polling indicated.
What Went Wrong?
It isn't entirely obvious why this bias occurred in both 2016 and 2020 polling, but we can make some good guesses.
The most plausible theory is that Trump supporters and Republicans disproportionately refuse to answer political polls, thereby omitting their support even after adjustments for demographic balance. This would imply a reduced response rate of about ten to twenty percent by Republicans relative to non-Republicans in the same demographic categories.
This could be because Trump supporters lied to pollsters (e.g. to "screw with the mainstream media") whom they distrusted and actively disliked as a group. But the "shy voter"/"dishonest polled person" hypothesis has been studied in past elections where polling was greatly at odds with the actual results, and those studies have repeatedly disfavored that explanation.
It is more plausible that Trump supporters were simply significantly less likely to respond to pollsters (whose response rate is low at the best of times, but usually not systemically biased with response rates being higher for one group than another in ways that cannot be demographically adjusted) for reasons causally related to being Trump supporters and to seeing the media establishment and academic and "expert" establishment that does polling as a source of "fake news" and not a group of people they want to cooperate with eagerly.
As one news analysis explained:
The 2020 polling error “matches the pattern of the 2016 error really
well, so there really does seem to be something wrong here,” explained
G. Elliott Morris, a data journalist who runs the Economist’s election
forecast, during a Wednesday postmortem on the “Science of Politics”
podcast. “It’s not just two random polling errors.” . . .
Researchers have largely ruled out the idea of “shy Trump voters” who
lie to pollsters and say they’re undecided or that they favor someone
else when they really favor Trump. But it’s possible, Grossman and
Morris speculated, that pro-Trump, non-college-educated whites are
simply less inclined to pick up the phone or participate in polls than
pro-Biden, non-college-educated whites.
Why? Because the pro-Trump cohort also tends to have less “social
trust” — i.e., less “trust in other people or institutions,” as Morris
put it. Spurred by Trump’s “fake news” mantra, participating in polls
may have itself become politicized. When overall response rates are as
low as 4 percent, this could skew the results against Trump in places
like the Rust Belt or Texas.
A similar dynamic may have also made it seem like more Republicans
were flipping from Trump to Biden than ultimately did — again, because
pro-Trump Republicans may be less inclined than pro-Biden Republicans
to answer a pollster’s call or participate in an online survey.
Scientific American magazine made a similar point in article:
Considering that most of the polls overestimated Biden’s lead, is it
possible pollsters were simply not adequately reaching Trump
supporters by phone?
David Shor, a data analyst [who was formerly head of political data
science at the company Civis Analytics], recently pointed out the
possibility that people who respond to polls are not a representative
sample. They're pretty weird in the sense that they’re willing to pick
up the phone and stay on the phone with a pollster. He gave evidence
that people are more likely to pick up the phone if they’re Democrats,
more likely to pick up under the conditions of a pandemic and more
likely to pick up the phone if they score high in the domain of social
trust. It’s fascinating. The idea is that poll respondents score
higher on social trust than the general population, and because of
that, they’re not a representative sample of the population. That
could be skewing the results.
This is also related to the idea that states with more QAnon followers
experienced more inaccurate polling. The QAnon belief system is
certainly correlated with lower social trust. And those might be
people who are simply not going to pick up the phone. If you believe
in a monstrous conspiracy of sex abuse involving one of the major
political parties of the U.S., then you might be paranoid. One could
not rule out the possibility that paranoid people would also be
disinclined to answer opinion polls.
We can even quantify that difference in likelihood of responding to the polls at 5-6 percentage points less likely to respond, all other things being equal, and probably a bit more, due to demographic adjustment of poll results in favor of historically Trump supporting demographics (perhaps 10 percentage points). This isn't such a huge difference in response rates as to be highly notable to someone doing polling, since response rates are low across the board. But it is enough to make a significant difference because the gap in response rates was so pervasive and so tightly tied to partisan leaning.
