7

Polls leading up to the election showed Biden with a commanding lead, averaged between 7 and 9 points depending on your source. But the result was Biden by 4.5.

I have a theory that some Republicans may have told a pollster they were planning to vote for Biden, but ultimately didn't even though they genuinely thought they would. There were reports of Republicans who felt Trump was immoral and didn't want to vote for him. This was particularly evident in Utah. I mean people who thought they were going to vote for Biden but felt they could not do that because he was a Democrat, and reluctantly voted for Trump's reelection.

Is there objective evidence such as post-election polling that suggests this was the case?

7
  • 1
    It's not implausible that 3% of Rep.-leaning, presumably non-MAGA voters wanted to vote for Biden but felt that they couldn't? Dec 7, 2020 at 16:02
  • 3
    Leaving aside the speculation on voters' precise motivations, the question is basically, "Were the polls off because people changed their minds about how they were going to vote?", or more broadly, "Why were the polls off?" I think the broader question might be more fruitful, but either question could be answered with facts, assuming any such facts are available.
    – MJ713
    Dec 7, 2020 at 16:23
  • 6
    In terms of "popular vote", the polls were not so "off".
    – Déjà vu
    Dec 7, 2020 at 16:23
  • 2
    I made a few edits for clarity. Please let me know if you feel they were improper.
    – MJ713
    Dec 7, 2020 at 16:36
  • 1
    Worth adding to note: The subset of people who are suspicious of information theft tend to fall majority Republican in The United States. This may account for some (but by no means all) polling underestimates of the Republican Candidate.
    – GOATNine
    Dec 8, 2020 at 19:31

4 Answers 4

21

The Polling Systemically Overestimated Democratic Results At Unprecedented Levels

The polling in the 2020 election had systematic problems that caused them to under-measure support for Republicans and Trump to a greater extent than any past election in more than a decade (including the 2016 election which was the runner up), and to over estimate support in favor of Democrats and Biden, in particular

This is something that is called technically "bias" in a poll (i.e. a lean in one direction due to errors across the board) as opposed to mere "error" which is disparity from the predicted outcome which does not itself lean one way or the other in any particular case, but isn't terribly accurate. In relative terms, the polling was largely on target, but the polling results were consistently more Democratic leaning than the actual outcomes. (I do not suggest that any pollster tried intentionally to get inaccurate results that favored Democrats and believe that they sincerely did their best based upon all evidence available to me. And, polling averages from disparate pollsters with different agendas would tend to filter out that kind of issue.)

It cannot plausibly be explained as simply being within the normal margin of error (the citation is to my own blog and is restated in part below; the link is included because the link provides access to more of specific factual results).

The 538 post-election analysis in 2020 looks a lot at historical "error" rates but contains some critical flaws.

First, a lot of the historical data compares a single poll (in the Gallup era) or a single polling average (national polling) to the result, and not a full set of state by state polling averages.

Second, it ignores the elephant in the room, which is that in 2020 the vast majority of significant polling errors overestimated Democratic performance. Polling error looks at how much the polls differed from the actual result. If half the races overestimate Biden performance by 4 points and half the races overestimate Trump performance by 4 points, the average polling error is 4 points, but the average lean is 0 points. But, if all of the races overestimate Biden performance by 4 points, the average polling error is still 4 points, but the average lean is also 4 points, which is a much bigger concern. The average error in the 2020 polling was high but not unprecedented. But the magnitude of the partisan lean in those errors in 2020 was unprecedented for a wide basket of separate polling averages.

I took the polling averages in all 50 states and the District of Columbia in the Presidential race and in each Senate race, complied by the Five Thirty Eight blog and compared them to the actual election results myself.

In the Presidential race (50 state polls, D.C. and the national polling average) the average pro-Democrat lean was 4.42 percentage points (compared to a previous record of 3.2 percentage points in 2016 and a 2.6 percentage point record lean pre-Trump and post-1998).

The pro-Democrat lean in the U.S. Senate polling in 35 races in 2020 was an average of 6.99 percentage points (compared to a previous record since 1998 of 4.8 percentage points in 1998).

(The mean error was somewhat larger than the mean Democratic lean. But adjusted for the Democratic lean in 2020 polling, as determined after the fact, the residual error in 2020 was smaller than in 2016, which is what was expected before the election since historical factors that favor higher error rates were absent.)

Keep in mind that each of these data points compares the actual results to a polling average of many polls by different polling operations in each race, not to a particular poll by one pollster or even a series of polls by one pollster. The averages were compiled from hundreds of individual polls, all recent and screened to keep out junk polling operations. The averaging method naturally makes the average more robust by screening for "house bias" to a great extent, reducing the expected margin of error relative to any individual poll, and screening out much the potential random sampling error. Essentially all of the lean towards one party or another has to be explained by systemic polling industry-wide issues.

Of 51 state polls in the Presidential race, the national Presidential rate polling and polling in 35 U.S. Senate races, a total of 87 polling averages, 79 (91%) erred in favor of Democrats.

Of the 87 polling averages, 54 (63%) were off by more than 4 percentage points (easily more than one standard deviation for a weighted average of election polls), with 98% of these large errors in favor of Democrats and the one large error in favor of a Republican coming in a Senate race without a Democrat running.

If the polls were not biased, you would expect a mean error in favor of Democrats of close to zero for 87 polling averages and mean errors overall of about half what we saw, with about equal numbers of polls biased in favor of Democrats and of Republicans both in terms of any errors, and in terms of large errors (about 43-44 errors for each party), and 28 or fewer large errors (about 14 errors for each party).

This level of polling bias is bigger than any election cycle polling bias in the 1998-2018 time period and about 33% greater than the next runner up in the Presidential races and 100% worse in the Senate races, than the next runner up, which was the 2016 Presidential election, which was biased in the same direction, for what in hindsight looks like the same reasons.

enter image description here

(538 source in 2018 article)

A lean like the one shown in these charts (which the lean in 2020 being record high) comes from only one source: non-random, systemic sampling errors. In 2016 and 2020 that non-random sampling error strongly appears to have been Trump-centric (although it might persist beyond him if attitudes towards surveys remain partisan). In other years, it was less severe and had different sources.

This is particularly notable because a priori the expectation was that polling error would be lower than in 2016 and would less strongly lean towards Democrats.

Years with high polling error are frequently associated with either lots of undecided voters (the New York Times flagged this as a key factor in polling errors in 2016) and/or a strong third-party performance. But in 2020, there were far fewer undecided voters than in 2016, and third-parties were not performing as well leading up to the election (or in the electoral outcomes themselves) as in 2016.

Pollsters across the board made efforts to increase cell phone coverage in their polling and to adjust the raw data towards demographics whose turnout had been underestimated in 2016. See also here.

High turnout also usually favors Democrats and high turnout of Democratic early and mail in voters in 2020 relative to any prior year (in part due to COVID-19) suggested that turnout models used to adjust raw polling results might underestimate Democratic candidate performance.

This isn't just my analysis. For example, an interview with Nick Beauchamp, assistant professor of political science at Northeastern likewise discounted the notion that it was just normal error:

There are a number of possible explanations, Beauchamp says. One is that specific polls undercounted the extent to which certain demographics—such as Hispanic voters in specific states—shifted toward President Trump.

Another is that, just as in 2016, polls undercounted hard-to-reach voters who tend to be less educated and more conservative. Beauchamp is less convinced that “shy” Trump voters deliberately misrepresented their intentions to pollsters.

“Whatever the cause, it has huge implications not just for polling, but for the outcome of the presidential and Senate elections,” Beauchamp says. “If the polls have been this wrong for months, since they have been fairly stable for months, that means that campaign resources may have also been misallocated.” . . .

This year’s polling errors, Beauchamp said, were “enormous” even compared to 2016, when polls failed to predict Trump’s defeat of Democratic nominee Hillary Clinton.

Indeed, just days before the presidential contest, FiveThirtyEight founder Nate Silver predicted that Biden was slightly favored to win Florida and its 29 Electoral College votes. The race was called on election night with Trump comfortably ahead. . . .

“I think Nate Silver and the other pollsters are saying ‘Well that’s just within the bounds of systematic polling,’ but it seems awfully large to me for just that,” Beauchamp says.

The Possibility Of Unexpectedly High GOP Turnout Isn't An Explanation That Can Explain This Systemic Error

If the polling error was due to more post-polling decisions to go to the polls made by Trump supporters than Biden supporters, this would show up in levels of partisan turnout. Turnout was expected to be high in advance of the election. And, while turnout was higher in the 2020 election as a percentage of the eligible voter population (VEP) since the year 1900, turnout was high for Democrats as well as for Republicans (whose turnout varies much less from year to year than Democratic turnout does). There just isn't good evidence of this kind of turnout disparity large enough to account for the over performance of both Trump and Republican U.S. Senate candidate, relative to the polling. Notably, GOP turnout was not record high in many of the swing states where polling was badly off. For example, yes, white Evangelical Christian voters turned out in phenomenal numbers. But this happens every year and was expected.

Put another way, the demographic balancing done in the polls in 2020 based upon the shortcomings of the 2016 polling basically got it right. The predicted demographic mix of voters on election day and the actual demographic mix of voters on election day weren't seriously and systemically off. Instead, what appears to have happened is that almost every demographic polled had more Trump supporters on election day than polling indicated.

What Went Wrong?

It isn't entirely obvious why this bias occurred in both 2016 and 2020 polling, but we can make some good guesses.

The most plausible theory is that Trump supporters and Republicans disproportionately refuse to answer political polls, thereby omitting their support even after adjustments for demographic balance. This would imply a reduced response rate of about ten to twenty percent by Republicans relative to non-Republicans in the same demographic categories.

This could be because Trump supporters lied to pollsters (e.g. to "screw with the mainstream media") whom they distrusted and actively disliked as a group. But the "shy voter"/"dishonest polled person" hypothesis has been studied in past elections where polling was greatly at odds with the actual results, and those studies have repeatedly disfavored that explanation.

It is more plausible that Trump supporters were simply significantly less likely to respond to pollsters (whose response rate is low at the best of times, but usually not systemically biased with response rates being higher for one group than another in ways that cannot be demographically adjusted) for reasons causally related to being Trump supporters and to seeing the media establishment and academic and "expert" establishment that does polling as a source of "fake news" and not a group of people they want to cooperate with eagerly.

As one news analysis explained:

The 2020 polling error “matches the pattern of the 2016 error really well, so there really does seem to be something wrong here,” explained G. Elliott Morris, a data journalist who runs the Economist’s election forecast, during a Wednesday postmortem on the “Science of Politics” podcast. “It’s not just two random polling errors.” . . .

Researchers have largely ruled out the idea of “shy Trump voters” who lie to pollsters and say they’re undecided or that they favor someone else when they really favor Trump. But it’s possible, Grossman and Morris speculated, that pro-Trump, non-college-educated whites are simply less inclined to pick up the phone or participate in polls than pro-Biden, non-college-educated whites.

Why? Because the pro-Trump cohort also tends to have less “social trust” — i.e., less “trust in other people or institutions,” as Morris put it. Spurred by Trump’s “fake news” mantra, participating in polls may have itself become politicized. When overall response rates are as low as 4 percent, this could skew the results against Trump in places like the Rust Belt or Texas.

A similar dynamic may have also made it seem like more Republicans were flipping from Trump to Biden than ultimately did — again, because pro-Trump Republicans may be less inclined than pro-Biden Republicans to answer a pollster’s call or participate in an online survey.

Scientific American magazine made a similar point in article:

Considering that most of the polls overestimated Biden’s lead, is it possible pollsters were simply not adequately reaching Trump supporters by phone?

David Shor, a data analyst [who was formerly head of political data science at the company Civis Analytics], recently pointed out the possibility that people who respond to polls are not a representative sample. They're pretty weird in the sense that they’re willing to pick up the phone and stay on the phone with a pollster. He gave evidence that people are more likely to pick up the phone if they’re Democrats, more likely to pick up under the conditions of a pandemic and more likely to pick up the phone if they score high in the domain of social trust. It’s fascinating. The idea is that poll respondents score higher on social trust than the general population, and because of that, they’re not a representative sample of the population. That could be skewing the results.

This is also related to the idea that states with more QAnon followers experienced more inaccurate polling. The QAnon belief system is certainly correlated with lower social trust. And those might be people who are simply not going to pick up the phone. If you believe in a monstrous conspiracy of sex abuse involving one of the major political parties of the U.S., then you might be paranoid. One could not rule out the possibility that paranoid people would also be disinclined to answer opinion polls.

We can even quantify that difference in likelihood of responding to the polls at 5-6 percentage points less likely to respond, all other things being equal, and probably a bit more, due to demographic adjustment of poll results in favor of historically Trump supporting demographics (perhaps 10 percentage points). This isn't such a huge difference in response rates as to be highly notable to someone doing polling, since response rates are low across the board. But it is enough to make a significant difference because the gap in response rates was so pervasive and so tightly tied to partisan leaning.

11
  • 1
    2012 could be argued as biased against Democrats using the same logic. Obama overperformed RCP's expectation by a similar margin. Dec 7, 2020 at 17:08
  • 1
    @NumberFile It is fair to say that the errors were so much greater in 2016 and 2020 because the Trump message was such that it particularly impacted polling response rates in a differential manner that prior GOP candidates did not evoke.
    – ohwilleke
    Dec 7, 2020 at 17:10
  • 1
    There is a difference bewteen "biased" and "badly biased." The hardest part of political polling is obtaining a representative sample of likely voters. Pollsters never succeed in getting that perfectly right. There was nothing exceptionally or inexplicable bad about how they did in this election. The word "bias" also suggests the wrong thing to people who aren't used to science and statistics. In this context, it simply describes a systematic error that the pollsters did their best to get rid of, but couldn't reduce to zero. It doesn't mean they were prejudiced or wanted to misreport results
    – user5526
    Dec 7, 2020 at 17:54
  • 3
    @ohwilleke: It's not clear to me from your answer quantitative measure of bias you're using, what you think it came out to be, and how big you think that figure would have to be to qualify as "badly biased." Also not clear to me what numbers you're using for the previous elections you refer to. And in general, this is a weird, fringe interpretation.
    – user5526
    Dec 7, 2020 at 18:40
  • 2
    Excellent answer at calling the cat a cat. It's one thing to note that errors are within error margins. It's another to note that they are all heading only one way. Biden Trump should have been a walkover according to the polls for months. Instead it ended up being a long hard slog. Note also that the word bias has a specific technical meaning for polls and stats, not to be confused with the general use of the term. This should be obvious to all but in a time when Chavez is claimed to have interfered in a 2020 election, it bears repeating, lest the paranoia of some gets triggered. Dec 8, 2020 at 18:18
17

In short, we don't know why polls were off with a fair argument made on the polls being 'good enough'.


In many ways, the polling error was within acceptable margins. I've been listening and reading the 538 overviews of polls for this year's election, and they pretty much say that there was no big problem (and that the result was not that close) but numerous small problems (as one might expect).

One of the 538 recent podcasts notes that as poll respondent numbers have dropped quite a bit compared to a few decades ago, the people who are responding are likely to not be representative of the overall voter population. This podcast also noted the importance of local knowledge in polling, incl. that these days phone area codes don't work as well as in the past because someone in Ohio may be using a NY area code and vice versa, which would affect the results.

And, of course, COVID-19 likely had some effect on the accuracy of the polls, but this is really difficult to quantify.

Other ideas regarding the results—but which really can't be tested to verify how much of an effect they had—include social alienation:

There are a number of possible explanations for this, and no definitive answers, but one thing I’ve come across in my public opinion research is that the share of Americans who are more socially disconnected from society is on the rise. And these voters disproportionately support Trump.

The idea that some of Trump’s supporters are more likely to be disconnected from civic life is hardly a new one. ... There was a large swing to Trump among white voters who had low levels of social trust — a group that researchers have found is also less likely to participate in telephone surveys.


The 'shy Trump voter' theory is also quite popular. However, 538 generally argues against the 'shy Trump voter' having had much of an effect:

First, the “shy Trump” theory relies on the notion of social desirability bias — the idea that people are reluctant to reveal unpopular opinions. ... But actual election results indicate that the opposite happened: Trump outperformed his polls by the greatest margin in red states, where he was quite popular. ...

The second reason to be skeptical of the “shy” theory is that Republican Senate candidates outperformed their polls too. ...

Third, Trump didn’t outperform his polls with the specific group of voters who research showed were most likely to hide their support for his candidacy. ...

Finally, Trump’s own pollsters told us that there weren’t many shy Trump voters by Election Day.

It would still add another level of uncertainty even if it was happening on a very small level.


There's the additional question of whether the high turnout as experienced in the election caused some of the poll biases. I've not seen an in-depth analysis of this, but it's been noted in some specific cases. The result has generally been that the higher turnout helped Trump in some areas and Biden in others:

Biden won these two states in large part by improving upon Democrats’ performance in each state’s metro area. For instance, he won Maricopa County, which includes Phoenix and usually makes up 60 percent of Arizona’s total vote share, by 2 points. He also surged in the Atlanta metro area...

But Trump still piled up a vote haul in Texas’s extensive rural areas and mid-sized cities large enough to overcome Biden’s improvement in the state’s major metro areas. ...

Given both candidates benefitted from this in different areas, I'm not certain a reliable quantification of this effect has been done yet—but unlike many of the other biases, this should be possible given we have an idea of how each voter group votes.

13
  • 9
    538 is just not right when it argues that polling error was within acceptable margins. The data simply does not support that conclusion. The error was unprecedentedly high and was decisively one direction which is not what normal polling error does.
    – ohwilleke
    Dec 7, 2020 at 17:12
  • 13
    @ohwilleke - Do you have statistics to back that up? Any single data point is always going to miss in exactly one direction. If you consider this year as a single data point, then "decisively one direction" is perfectly normal. If you consider it as 50 data points, then yes, it's more unusual, but it's not 50 independent data points. The same pollsters poll different states, and thus all the polling error is going to be correlated.
    – Bobson
    Dec 7, 2020 at 17:34
  • 5
    @Bobson I have 87 data points, each drawn from an 538 correctly weighted average of polls that should have a lower margin of error than any individual poll, even if you assume long tails as they do in their predictions.
    – ohwilleke
    Dec 7, 2020 at 17:41
  • 6
    @Bobson Clearly, systemic bias must have been at play. Virtually all polls underestimated Trump - that just can't have been coincidence or randomness. Dec 7, 2020 at 17:53
  • 5
    @ohwilleke - According to 538 a four point difference is an average miss. Per the numbers in the question, the polls missed by 3-4.5 points. I'd consider that "normal" not "unprecedentedly high". There were definitely outliers this year (like Florida), but there were also states that were very close to their numbers (Georgia). So why do you consider that high, and what would you consider to be a normal miss? Edit: scratch that last question. I'll go read your answer.
    – Bobson
    Dec 7, 2020 at 18:39
3

It's certain that some voters change their minds at the last minute. However the only thing we can find out from standard-style opinion polls and election results is the net effect, which is the difference between the number of last-minute switches from A to B and the number of last-minute switches from B to A. Even then, there is no easy way to tell whether a difference between polls and election results should be interpreted as a last-minute change of heart or (more likely) standard polling errors, which often have to do with things like likely voter models and difficulties in getting a representative sample of voters who are willing to answer a poll.

However, there are tracking polls, such as the one done by Morning Consult in 2000. Here they take the same group of people and get them to repeatedly register their voting intentions over a long period of time. These polls generally showed that the 2020 presidential election was extremely static. Almost everyone made up their mind long before the election. In this respect it was completely different from 2016.

In general, the 2020 polls were about as accurate as you should expect an opinion poll to be. They had a huge, notable miss in Miami-Dade County. Other than that, the polls got almost all states right, with no huge upsets. States like Florida were known to be in doubt based on the polls. States like Pennsylvania and Texas were not in so much doubt, and were called correctly based on the polls.

2

It's possible that Democratic engagement soared, leading to more poll respondents who leaned heavily Democrat

So the basic story is that, particularly after Covid-19, Democrats got extremely excited and had very high rates of engagement. They were donating at higher rates, etc., and this translated to them also taking surveys, because they were locked at home and didn’t have anything else to do. There’s some pretty clear evidence that that’s nearly all of it: It was partisan non-response. Democrats just started taking a bunch of surveys [when they were called by pollsters, while Republicans did not].

And later

It used to be that once you control for age and race and gender and education, that people who trusted their neighbors basically voted the same as people who didn’t trust their neighbors. But then, starting in 2016, suddenly that shifted. If you look at white people without college education, high-trust non-college whites tended toward [Democrats], and low-trust non-college whites heavily turned against us. In 2016, we were polling this high-trust electorate, so we overestimated Clinton. These low-trust people still vote, even if they’re not answering these phone surveys.

In other words, people didn't change their minds for Biden, they simply didn't answer surveys as often as Democrats. That helps to explain wildly wrong polls like this Washington Post/ABC poll from Oct 28, 2020

Former vice president Joe Biden leads President Trump by 17 points in Wisconsin, according to a new Washington Post-ABC News poll out this morning.

Trump lost Wisconsin by 0.3%. This also helps to explain other down-ballot polling problems where Republicans won races polling showed them soundly losing. One such place was Maine, where Susan Collins never lead a single poll, yet Collins won by nearly 8 points

1
  • That is definitely a possibility. I feel like the answer is mostly no. Dec 9, 2020 at 15:52

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .