Phenomenon where politically incorrect candidates do better in actual elections than in polls?

Question

I've long hypothesised that a certain phenomenon may exist, and I wonder if there's any actual research on the subject.

I'm asking for research, statistics, etc. on a macro level. However, I think it's easier if I describe the concept on a micro-level, using Bob as an example. This is only an illustration of the concept, it's not the actual question:

Bob has political views that are not considered politically correct by society at large. Some of them are considered racist by many. Some of the proposals he supports are considered unrealistic, populist or very detrimental by mainstream outlets. The views Bob holds are often warned against as racist, dangerous, etc. by media, academics, large-party politicians, etc.

Now, in spite of these views being seen as politically incorrect, Bob still holds them. However, during lunch hour, when his colleagues talk negatively about those very views, he does not join the discussion. Instead, he pretends he agrees with his colleagues, or at least, he keeps his mouth shut. Not only is he afraid of being socially excluded at work, on a certain level he is also internally embarrassed and somewhat conflicted about the views. So he keeps these views secret, hiding them from his family, friends and even strangers. He also feels embarrassed within himself over this. However, in spite of the embarrassment, his predominant feeling is that the views are correct.

There is an upcoming election, and one of the candidates or parties promotes many of the same "politically incorrect" views that Bob holds.

One day, Bob is called up by a pollster. They ask him who he intends to vote for in the upcoming election. Even though Bob doesn't know this person, he still doesn't like to reveal his secret views to another human being. (Even a stranger.) So, he says that he is going to vote for another party/candidate. Or he declines to answer.

However, when the actual election comes, he can vote without saying it to another human being. He can just drop his vote in the ballot box. So he does so, for the candidate/party he actually likes. (The "politically incorrect" one.)

So, what I'm asking is: Is there research or statistics showing whether controversial or "politically incorrect" candidates/parties tend to do better or worse in actual elections than in polls?

I realize defining what a "politically correct" or controversial candidate/party means is somewhat subjective. But most political research involves some subjective judgements.

Reason for bounty: I am happy with the currently accepted answer, and tend to support its ideas personally.

However, during a course on philosophy of science at Oslo Metropolitan University, my lecturer was discussing this exact topic. I brought up social desirability bias and she seemed almost dismissive of it, saying pollsters and statisticians have ways to account for that, and she brought up some other theories (such as respondents in polls not actually matching the voting population. For example, the voters of Candidate A may really dislike polls, or not have telephones).

To be honest, I am not sure I agree with my lecturer, but I still want to hear some various perspectives and get more responses to this question.

I am really happy with the accepted answer, and personally it rings true to me. The purpose of the bounty is just as the notice says, to "draw attention". I am open to other answers and theories, it would be interesting.

Answer 1

One term for this effect is "Social Desirability bias"

Social desirability bias is a social science research term that describes the tendency of survey respondents to answer questions in a manner that will be viewed favorably by others. It can take the form of over-reporting "good behavior" or under-reporting "bad", or undesirable behavior. The tendency poses a serious problem with research with self-reports, especially questionnaires. This bias interferes with the interpretation of average tendencies as well as individual differences.

Kyle Dropp, the director of polling company "Morning Consult", explicitly attributed this effect as a likely reason for discrepancy between polls and vote counts for Donald Trump (who's a poster child for "politically incorrect candidate" and, I'm willing to bet, an inspiration for your question) in 2016 Republican Primaries in US, as discussed in my answer here.

An interesting data for this comes from comparing person-conducted polls compared to online surveys (quotes from LA Times's surprisingly insightful article on the topic: "Polls may actually underestimate Trump's support, study finds"):

.. confirmed that "voters are about six points more likely to support Trump when they’re taking the poll online then when they’re talking to a live interviewer,” said Dropp.

Some significant number of Trump supporters, especially those with college educations, are "less likely to say that they support him when they’re talking to a live human” than when they are in the “anonymous environment” of an online survey, said the firm's polling director, Kyle Dropp.

The most telling part of the experiment, however, was that not all types of people responded the same way. Among blue-collar Republicans, who have formed the core of Trump's support, the polls were about the same regardless of method. But among college-educated Republicans, a significant difference appeared, with Trump scoring 9 points better in the online poll.

UPDATE 2016/11/09: FiveThirtyEight in their post-election analysis also noted similar effect (thought not every pollster seems to agree) in general election:

Several pollsters rejected the idea that Trump voters were too shy to tells pollsters whom they were supporting. But James Lee of Susquehanna Polling & Research Inc. said his firm combined live-interview and automated-dialer calls, and Trump did better when voters were sharing their voting intention with a recorded voice rather than a live one.

Women who voted for Trump might have been especially reluctant to tell pollsters, said David Paleologos of Suffolk University. The USC Dornsife/Los Angeles Times poll corroborated that: “Women who said they backed Trump were particularly less likely to say they would be comfortable talking to a pollster about their vote.”

---

A related effect (or rather, a particular specific instance of Social desirability bias) was mentioned in the comments by @Sjuan76: Bradley effect. Quoting Wiki summary:

The Bradley effect (less commonly the Wilder effect) is a theory concerning observed discrepancies between voter opinion polls and election outcomes in some United States government elections where a white candidate and a non-white candidate run against each other. The theory proposes that some voters who intend to vote for the white candidate would nonetheless tell pollsters that they are undecided or likely to vote for the non-white candidate. It was named after Los Angeles Mayor Tom Bradley, an African-American who lost the 1982 California governor's race despite being ahead in voter polls going into the elections.

The Bradley effect posits that the inaccurate polls were skewed by the phenomenon of social desirability bias. Specifically, some white voters give inaccurate polling responses for fear that, by stating their true preference, they will open themselves to criticism of racial motivation. Members of the public may feel under pressure to provide an answer that is deemed to be more publicly acceptable, or 'politically correct'. ...

Some analysts have dismissed the theory of the Bradley effect, or argued that it may have existed in past elections, but not in more recent ones, such as when Barack Obama was elected and reelected President of the United States in 2008 and 2012 respectively. Others believe that it is a persistent phenomenon.

Answer 2

Based on subsequent controlled studies, the shy-Trump-voter effect was overblown. The polls in question had other issues, sampling mainly. A more recent review concludes that

Generally, there is little evidence that voters lying about their vote intention (so-called ‘shy’ voters) is a substantial cause of polling error. Instead, polling errors have most commonly resulted from problems with representative samples and weighting, undecided voters breaking in one direction, and to a lesser extent late swings and turnout models.

And in more detail from the review:

Given the secrecy of the ballot, it is difficult to prove whether people lie about how they will vote. There is some evidence to suggest it happens on at least some occasions. In the US, the ‘Bradley effect’ whereby polls overstated support for African-American politicians (due to white respondents being reluctant to say they would vote for a different candidate) appears to have been true in the 1980s and early 1990s, though it has since disappeared (Hopkins 2015). As with the ‘late swing’ excuse, ‘shy’ voters are more often accused than guilty (Coppock 2017; Durand et al. 2001, 2002; Kennedy et al. 2017; Mellon and Prosser 2017a; Sturgis et al. 2016). ‘Shy Tories’ – supporters of the Conservative Party in the UK – are the canonical explanation for the 1992 polling miss in Britain. There is some evidence that people who did not disclose their vote intention prior to the 1992 election but reported voting in recontact surveys were disproportionally Conservative (Jowell et al. 1993b), though whether this represents ‘shy’ voters or late deciders swinging in one direction is unclear, and the extent to which people lied to pollsters has been disputed (Crewe 1993; Worcester 1996).

Shy Tories were suggested to be behind the 2015 polling miss (Singh 2015), but looking at differential vote by non-disclosers during the campaign, the geographic distribution of Conservative support and vote intention reporting, a question ordering experiment, and modelling the likely vote choices of non-disclosers, Mellon and Prosser (2017a) found no evidence of a Shy Tory phenomenon, a conclusion later confirmed by the British Polling Council inquiry into the polling miss (Sturgis et al. 2016). Some researchers also suggested that people lying to pollsters about their likelihood of turning out to vote was a factor in the polling error (Whiteley 2016). Examining this question, however, Mellon and Prosser (2017a) found that although people over-reported their likelihood of voting and having voted, over-reporting was uncorrelated with vote choice, meaning the errors in vote share estimates cancelled out.

‘Shy Trump’ supporters were also suggested to be behind Trump’s unexpected victory (Anderson 2016). However, there is no evidence supporting this. Using a list experiment – a technique designed to elicit accurate reporting of socially undesirable behaviour – Alexander Coppock (2017) found no evidence of Shy Trump voters. Likewise, the AAPOR report into the 2016 US polls compared presidential vote intentions to Republican support in down ballot races and found nothing to suggest a Shy Trump effect (Kennedy et al. 2017).