Is it possible to have a 22 point difference between polls taken over the same time period?

Question

So I checking the state of the Democratic Nomination polls today and I saw this head scratcher:

• Quinnipiac 10/17 - 10/21 21 28 15 10 5 ... Warren +7
• CNN 10/17 - 10/20 34 19 16 6 6 ... Biden +15

Both polls were taken over the same time period. Both polls have a large sample size of 1587 and 1003 respondents respectively. Theoretically the margin of error for each poll is listed at +/- 3.7% to 4.6% and yet there is a 22% difference! How is this possible?

I know that the margin of error is only 95% accurate so I wanted to simulate this properly. I wrote a quick python script to do a poll over a simulated infinite population that voted according to the RCP averages. After 1 million trials, the biggest difference between polls I ever got was 14% with poll A showing Biden +14% and poll B showing a statistical tie.

Is it possible the polling institutions are being dishonest or fudging the numbers somehow?

import random

def get_poll(sample_size):
    '''Returns Biden's polling advantage vs Warren over the sample size'''
    warren = 0
    biden = 0

    for person in range(0, sample_size):
        r = random.random()
        if r < .218:
            warren += 1
        elif .218 < r < .490:
            biden += 1
    return biden/sample_size - warren/sample_size

big = 0
for trial in range(int(1e6)):
    random.seed()
    a = get_poll(1587)
    b = get_poll(1003)
    spread = abs(a-b)

    if spread > big:
        big = spread
        print("\nPoll A: Biden advantage:", int(a * 100))
        print("Poll B: Biden advantage:", int(b * 100))
        print("Spread:", int(spread * 100))

    if not trial % 10000 and trial:
        print("Test number:", str(int(trial/1000))+'k')

Output:

Poll A: Biden advantage: 0
Poll B: Biden advantage: 14
Spread: 14

Answer 1

I'm not sure we can tell any better than NYT commented on this:

It is not clear why the CNN and Quinnipiac polls present such divergent results for Ms. Warren and Mr. Biden. But the polls, both considered to be of high quality and able to help candidates qualify for the next Democratic debate, add data points to a divide that has emerged in recent weeks: between surveys that show Mr. Biden with a commanding lead, and those that have him in a statistical dead heat with Ms. Warren.

A number of things can affect a poll’s results, including the wording of a horse-race question and the order that items are asked. In CNN and Fox News polls this year, respondents have typically been asked for their opinions on each of the Democratic presidential candidates, among other questions, before being queried about their vote preference.

In Quinnipiac’s polls — as well as those conducted by Monmouth University, which have also shown Ms. Warren climbing steadily — people have not typically been asked to evaluate the candidates one-by-one before giving their vote choice.

If such small differences in survey structure are indeed having an effect on results, it may reflect the fact that many respondents are not yet certain about their feelings.

The CNN poll released this week brought that point home, finding that a majority of Democratic voters who favored one candidate — 53 percent — said they could still change their mind about whom to support for the nomination.

“It makes me think that voters are not settled, that they’re still shopping for a candidate — that’s why you’re seeing some movement between polls,” Doug Schwartz, Quinnipiac’s polling director, said Thursday in a telephone interview.

My own observation using 538 summary is that Quinnipiac’s poll has put Warren in the lead since August, in all four Quinnipiac polls after August, to be more precise. So there's probably something in Quinnipiac's methodology that favored Warren recently (more than once), but even they probably don't know what.

Answer 2

It's not necessarily dishonest or fudging, but consider that in all likelihood, both polls asked 2590 unique people and no one repeated. Then consider that the polls are not likely asking the same questions across the board. The margin of error is for "If this test was duplicated with a sample sizes, you will get a percent value answering in favor of x and a value in favor of y."

Quinnipiac are likely not running the same questions, nor to the same people, nor will they get the same responses because of the name of the agency. Additionally, because those aren't the only two candidates, the margin of errors don't mean same thing and you're not reporting what score Quin. gave Biden or what score CNN gave Warren. Nor do we know if all options are on the table (could Quinn have more candidates on the list, while CNN omitted low performers?).

Again, back to what was asked, the wording and options of the poll could be different (Say, Quin. asks "Who will you vote for in the Primary?" while CNN asks "Would you support [insert candidate] if nominated for President?") In the former, you're picking the candidate you will vote for, while in the later you are voting for the candidate you may not have voted for but will still support if they ran.

Also, if the question is rating support (say strong disapproval, disapproval, neutral, support, strong support) they may aggregate positive answers and negative answers as the numbers for strong and not-strong will likely not be a high percent on their own, but when their powers combine, it can get much closer to a big cut. Some will even throw neutral into the pile, just too sure up that they're still being considered, and could go that way.

Additionally, are you accounting for the fact that the margin of error will still need to balance with the other responses, so that it comes from an even 100%? In the CNN poll, if Binden has the strongest possible support in the bounds of margin of error, that means the others will have to be subtracted from to make up the difference. Suppose that Biden was really +19.6% and the difference came out of the other candidates. This puts Biden much closer to 22% while the next highest candidate might go down so that the difference between them.

Suppose both polls erred against Warren to the Margin of error. Quin short changed Warren and she goes up to +10.7%, while CNN over hyped Biden and he goes down to 10.4%. This means your 22 gap went to just a 0.3 gap. And since both polls are independent, they only have to meet 100% internally. Externally, we can fiddle with the margins to see if both are close by reducing the higher candidate while boosting the lowest candidate. We can also max them out, or short them, though that still leaves a much wider gap (although Min Quin Warren and Min CNN Biden are a lot closer than Min Quin Warren and Max Quin Biden.). If we assume the Warren error happened in both, though, we can see the results are the same. If I had Quin Biden and CNN Warren, I could probably see that an error favoring Biden brings the numbers close as well).

Answer 3

In big oversimplfication:

Difference: 22 p.p.

one standard deviation (from confidence intervals):

• for Quinnipiac: 1.85 p.p.
• for CNN 2.3 p.p.

So you need both estimations to be off by 5.3 s.d. Chance of one such event (one sided) is one in 17 270 757. As we need them both to happen simultaneously, in simplification we're talking about an event that should happen for every 2.98*10^11 pairs of polls.

If you only were patient run one million times your program, each time with a milion samples, you should have get a few of such flukes. :D

So for practical purposes we eliminated chance of some random event. The default answer in such situations would be a total disaster in sample selection. However, as according to recent leak CNN management was not above pushing news that even its own staff considered as dubious or were effectively trying to handpick the Democratic candidate, I'd not rule out purposeful manipulation.