20

In November 2019, many polls (such as this ABC/Washington Post poll from November 5th) claimed that the Democratic candidates had double-digit leads over Trump. In the 2016 elections, however, polls suggested that Clinton would beat Trump, which turned out not to be the case.

Based on this history, were these polls trustworthy? Or could they have been biased or inaccurate?

9
  • 19
    As luck would have it, Nate Silver at 538 just published an article on the state of political polling which evaluates accuracy and sources of error and bias over time and in the most recent polling cycles: fivethirtyeight.com/features/the-state-of-the-polls-2019
    – divibisan
    Commented Nov 5, 2019 at 22:40
  • To answer your question you will have to specify which polls specifically you are talking about. Commented Nov 6, 2019 at 14:22
  • 22
    An aggregate of polls in 2016 showed that Clinton would probably win, but it would likely be close. A lot of pundits spun that into "Clinton will definitely win" and disappointed a lot of people when that didn't come to pass. As it happened, Clinton did win the popular vote by quite a bit, and in the 3 swing states Clinton needed, the margin of victory combined was only 70,000 votes. Taken as a whole, I'd say the polls were pretty damn accurate.
    – Seth R
    Commented Nov 6, 2019 at 16:52
  • 6
    I think you have to be clearer about just what you mean by "trustworthy". You could read that as either being fairly conducted and accurate to within their statistical limits, or in the Trumpian sense that all the pollsters are conspiring to present false results.
    – jamesqf
    Commented Nov 6, 2019 at 18:10
  • 5
    @Anoplexian Indeed, and when you take an aggregate of a bunch of polls, the margin of error gets even tighter than that. I dispute the whole notion of "the polls were wrong in 2016". No, people who didn't understand or ignored what the polls were saying were wrong. Reality actually bore out well within the tight margin of error the available data said it would. The uncertainty factor just didn't swing the way a lot of people assumed it would.
    – Seth R
    Commented Nov 7, 2019 at 20:49

4 Answers 4

57

Fully-specified polls are perfectly trustworthy in terms of what polls actually indicate. Their topline results are not very reliable.

Most people think of polls in various incorrect ways. The most common (and least meaningful) way is as a straight prediction of how people will vote on election day. Polls almost never try to indicate this. Even when hacks present them in a way that suggests that's what they show, they're usually not so honest about that.

Consider a poll indicating a 85% chance of result A (and this is already a vague prediction), and a 15% chance of result B. That result certainly suggests that we're more likely to see A, but also indicates a nonzero chance of B. If the outcome turns out to be B, that doesn't mean that the poll was wrong. It was the assumption that an 85% prediction of A meant that A definitely would happen that is wrong, and that's not something that the poll claimed. And that "85% chance of A" is already an odd way to think about a polling result, though it's possible that a poll could be designed to indicate something like that.

The correct way to interpret a poll is pretty involved-- you have to understand their sampling strategy, what assumptions get "baked in" to the information that comes out of the polls as a result, the margin of error, the response rate, the effect size the poll can detect based on its statistical power, and other factors.

Different groups definitely release polls in more of a public relations context than a mathematical or scientific one. Understanding how those polls were conducted can reveal any shenanigans undertaken during the polling process, but a bigger issue is that groups with a strong advocacy will only publicize polls that illustrate the result they want on people's minds.

Statistical information is generally internally consistent, and the fully specified protocol tells you exactly what the statistic means, and the context in which that meaning exists. If you are not permitted to review a poll's methodology, it is wise to be very skeptical of that poll. If you can review the methodology, then the poll indicates exactly what it was specified to indicate, with mathematical bounds on how precise that information can be.

So polls about expected 2020 federal election results can be very trustworthy for what they actually measure (which is usually something like "if the election were held tomorrow..."), and that information is not quite meaningless regarding the actual 2020 election results. But they are not very strongly predictive. Anyone telling you that the correct interpretation of their poll means a specific result will happen in November of 2020 is either lying, expressing themselves poorly, or is an incompetent pollster/interpreter of polls.


As a side-note, the national polls around the 2016 election were fairly accurate in terms of vote totals-- they were wrong about the distribution of votes with respect to electoral college votes, which happens to be more relevant to securing the Presidency. In-state poll results varied, but weren't off by that much that the polls should be considered worthless.

Polls deal with likelihoods based off of current, incomplete information. It is very, very rare that a poll is meant to provide (let alone capable of providing) a reliable point estimate or a prediction of a clear, specific binary outcome.

20
  • 30
    Also of note, the presidency isn't decided by a national vote, but 50 state elections. In addition to the uncertainty factors you've noted, state-by-state polling is very thin this far ahead of the actual ballot. Commented Nov 5, 2019 at 17:03
  • 5
    @jeffronicus 50 state elections plus the District of Columbia with 3 electoral votes.
    – doneal24
    Commented Nov 6, 2019 at 1:34
  • 6
    @Nacht-ReinstateMonica A fair amount of the public is at least somewhat educated on polling and statistics, which means there can be value in public consumption. As for mainstream news, it’s entirely reasonable for news outlets to have polls presented to the public by experts who explain what the poll actually shows, in what context and with which limitations and with how much certainty. Which is, by and large, what they claim to do and also what they, as far as I can tell, honestly try to do. A lot of pundits aren’t so great at it, so room for improvement, but not “doesn’t belong anywhere.”
    – KRyan
    Commented Nov 6, 2019 at 4:12
  • 4
    @KRyan I think you are strongly overestimating the general public. If you look at experimental results by Kahneman, Tversky, Ariel and others, it seems to be the case that not even highly educated professionals (medical are the ones I remember but I think others were studied too), are capable of properly interpreting probabilities. I strongly doubt that the general public can do better. From my experience most people (including myself as a person with a Masters in Mathematics) are incredibly poor at judging what kind of probability a given result has ...
    – DRF
    Commented Nov 6, 2019 at 12:24
  • 4
    @Nacht-ReinstateMonica I'd say that the general public's ability to correctly interpret polls is (counterintuitively) not an acute issue. The vast majority of people don't seem to interact with, or consume, polling results at all. They tend to rely on others to interpret the polls for them (sort of like relying on an auto mechanic or a physician) to give the correct meaning of a poll. If those interpreters are dishonest or incapable, that's a real problem, but not a problem with polling data. I think it's a good thing for the data to be available for anyone that wants to examine it.
    – Upper_Case
    Commented Nov 6, 2019 at 15:49
30

Don't mistake polled sentiments for election results.

In all likelihood, the next presidential election in the US will be decided by a few swing states. It will not matter how much the Democrats win in a deep blue state. It will not matter how much the Republicans win in a deep red state. All that matters is who wins states like Florida, Ohio, or Virginia (among a few others) by the thinnest of margins.

Donald Trump appears angry that Hillary Clinton actually got more votes than he did, but the fact is that under the US electoral system, Trump's 62,984,828 voters defeated Clinton's 65,853,514 voters. Because Trump got more electoral college votes.

2
  • 2
    See also: Nate Silver's commentary. "Election polls sometimes get the answer wrong — but they’re about as accurate as they’ve always been." And also this more recent article regarding the state of the polls. Commented Nov 6, 2019 at 16:04
  • 1
    The next presidential election may be decided by swing states, but we all know who's likely going to win based on history, not polls. And that's the sitting president. If he runs again and isn't shot in the meantime (a non-zero risk with US presidents history tells us) and global war doesn't break out, the chances of winning are way over 50%. That's how it goes. Regardless of polls and swing-states.
    – Mast
    Commented Nov 8, 2019 at 7:25
15

In addition to the caveat about the Electoral College, polls can change pretty fast. Just on Oct 30, another predicted Trump would win the popular vote, against an unnamed Democratic nominee, by a narrow margin. Even more noteworthy, the subjects gave an even higher rating to Trump's chance of actually winning the race (probably because people are aware of issues like the Electoral College and swing states):

enter image description here

The poll surveyed 1,000 registered voters across the country from October 23 to October 26 and has a margin of error of 3 percentage points. The findings of the poll were released on Wednesday, October 30.

And that poll is from a fairly reliable pollster (Suffolk University) as well (A- rating on 538).

Additionally, there can be large differences between well-rated pollsters on seemingly very similar questions.


And NYT had a much more sensible take (than the OP's question seems to suggest) on the latest polls (published Nov 4-5)

In national polls, Mr. Trump’s political standing has appeared to be in grave jeopardy. His approval ratings have long been in the low 40s, and he trails Mr. Biden by almost nine points in a national polling average. But as the 2016 race showed, the story in the battleground states can be quite different. Mr. Trump won the election by sweeping Michigan, Pennsylvania, Wisconsin, Florida, Arizona and North Carolina — even while losing the national vote by two points.

Democrats would probably need to win three of the six states to win the White House, assuming other states voted as they did in 2016 — an outcome that is not at all assured.

The Times/Siena results and other data suggest that the president’s advantage in the Electoral College relative to the nation as a whole remains intact or has even grown since 2016, raising the possibility that the Republicans could — for the third time in the past six elections — win the presidency while losing the popular vote.

There is a full year before Election Day, and a lot can change. [...]

enter image description here

enter image description here


AAPOR which did an extensive post-hoc evaluation of the 2016 polling (fiasco) noted that state-level polling, which matters a lot in the final outcome, was rather unreliable in 2016 in part due to poor sampling compesation and in part due to apparently last-minute decision making by undecieded voters.

National polls were generally correct and accurate by historical standards. [...]

[However] State-level polls showed a competitive, uncertain contest… …but clearly under-estimated Trump’s support in the Upper Midwest.

In the contest that actually mattered, the Electoral College, state-level polls showed a competitive race in which Clinton appeared to have a slim advantage. Eight states with more than a third of the electoral votes needed to win the presidency had polls showing a lead of three points or less (Trende 2016). As Sean Trende noted, “The final RealClearPolitics Poll Averages in the battleground states had Clinton leading by the slimmest of margins in the Electoral College, 272-266.” The polls on average indicated that Trump was one state away from winning the election.

Polls showed Hillary Clinton leading, if narrowly, in Pennsylvania, Michigan and Wisconsin, which had voted Democratic for president six elections running. Those leads fed predictions that the Democratic Blue Wall would hold. Come Election Day, however, Trump edged out victories in all three. [...]

Many polls – especially at the state level – did not adjust their weights to correct for the over-representation of college graduates in their surveys, and the result was over-estimation of support for Clinton. [...]

About 13 percent of voters in Wisconsin, Florida and Pennsylvania decided on their presidential vote choice in the final week, according to the best available data. These voters broke for Trump by near 30 points in Wisconsin and by 17 points in Florida and Pennsylvania.

While pollsters may be able to fix the sampling compensation issue for [less] educated voters, it's a lot less clear if they can do anything about the last-minute-decision voters.

7
  • @divibisan: my point is that unlike what the OP claimed (without a source) the NYT didn't interpret the poll as a sign that Trump will surely lose the 2020 election. I've made an edit to that effect. Commented Nov 5, 2019 at 22:05
  • 5
    The problem with that first poll is that it's just a generic Democratic candidate. The actual numbers are likely going to vary considerably depending on just who the nominee turns out to be. For instance, Warren or Sanders are likely to turn off a good many nonpartisan voters.
    – jamesqf
    Commented Nov 6, 2019 at 4:20
  • @jamesqf: Yes, I've actually read that somewhere, but I could not find the source again, and I did not dare mention it without a source (on a controversial topic like this). I think it was a WaPo or NYT article... Commented Nov 6, 2019 at 4:22
  • @jamesqf: From what I can recall, the article was illustrating the issue by showing that in the same poll the generic Democratic candidate was doing worse (vs Trump) than any of the three leading ones when named. What I could find [again] was an article showing a poll in which Hillary Clinton was doing worse than the trio (but still beating Trump in the popular vote), although that's not exactly the same issue. newsweek.com/… Commented Nov 6, 2019 at 4:33
  • 3
    Upvoting this solely for the NYT article reference. That makes this the only answer here so far that's aiming at the right thing (states, not national).
    – T.E.D.
    Commented Nov 6, 2019 at 16:51
4

Are the polls trustworthy? For most of them, yes, they're pretty reliable with actual results usually falling within their stated margins of error.

However, whether these results will actually translate into election night results is an entirely different question. We're still an entire year away from election night. We're still likely several (and minimally a few) months away from knowing who will even be running against him. So, while the polls may be relatively accurate in terms of showing what the nationwide popular vote would be if suddenly, unbeknownst to either party, the election were moved to today, that doesn't necessarily translate well into what will happen a year from now.

Right now, the Republicans are mostly sitting back and letting the Democratic candidates duke things out amongst themselves. Once a nominee has been selected, though, things will likely change rapidly on that front. The strategy Republicans take on the Presidential election will largely depend on who the Democrats end up nominating. They'd need a rather different strategy against a more moderate Democrat like Biden than they'd need against a far-left one like Sanders or Warren.

For comparison of how quickly things can change once the general election gets underway, especially when an incumbent President is running, I'd recommend taking a look at the last time this has happened.

At this point in his Presidency, Obama's Gallup approval rating (42% on Nov 5, 2011) was almost equal to Trump's current one (41%.) By Nov 4, 2012, it was 52% and Obama won re-election by a significant margin.

Of course, it's also entirely possible that Trump's favorability and approval ratings could plummet over the next year (especially with all of the corruption investigations going on) and he could lose by a landslide. A year is a long time in politics and there are many things that could happen. So, while the current numbers are probably accurate measures of what they're actually measuring (current sentiments,) they're not necessarily a great barometer of how an election will turn out in a year.

4
  • The Quinnipiac and CNN polls just had a 22 point difference between them. That's well outside the margin of error. In fact it's outside the margin of error more than 6 times over. Commented Nov 9, 2019 at 17:21
  • @SurpriseDog Which CNN and Quinnipiac polls? The most recent Trump vs. Biden ones that I see listed on RCP have spreads that are only a point apart. Trump vs. Warren was identical and Trump vs. Sanders differed by 2 points.
    – reirab
    Commented Nov 9, 2019 at 18:46
  • politics.stackexchange.com/questions/47054/… Commented Nov 9, 2019 at 19:32
  • @SurpriseDog Oh, you're talking about the primary poll. Honestly, I'd guess Fizz's guess is correct that the questions being asked before it in CNN's poll vs. none asked before it in Quinnipiac's made a difference, especially with Warren not being as well known to a lot of people who don't follow politics closely. Especially with those polls being before several of the recent events that have brought the different Democratic candidates' views and plans to a wider audience.
    – reirab
    Commented Nov 10, 2019 at 7:02

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .