Define a polling fail s.t.

  1. a particular candidate C, running in ...
  2. a particular election E, s.t. ...
  3. pre-E polls (collectively, averaged) predict C to achieve vote share S0, but ...
  4. results from E give C vote share S1, s.t. ...
  5. S0 differs significantly from S1 ((S0 << S1) || S0 >> S1))

Observers of US politics often find instances of polling fails s.t.

[type A] (C is generally perceived to be "left of center"[1]) && (S0 >> S1)

i.e., a US correlate of the UK "shy Tory factor". A recent US instance (as of 4 Nov 2020) is the 2020 election for US president. There are however cases where "politically-opposite" behavior has been observed, s.t.

[type B] (C is generally perceived to be "right of center"[2]) && (S0 >> S1)

such as the 2012 US presidential election, where the "left-of-center" candidate outperformed his pre-election polls.[3].

My question is: have polling fails as defined here been studied both quantitatively and with regard to left-right political orientation/spectrum? I'm looking for pointers to studies that (e.g.) classify polling fails in a manner similar to my type A and type B (above), and which then compare the numbers of each.

[1]: In US political classification, "left of center" typically includes Democrats as well as persons labeled "liberal" or "progressive." (This is a characterization, not an equation!)
[2]: In US political classification, "right of center" typically includes Republicans as well as persons labeled "libertarian" or "conservative." (Ditto.)
[3]: It's also a case of a US-minority-ethnicity candidate reversing the "Bradley effect", though there are certainly more recent instances (as of 4 Nov 2020) combining both the Bradley and shy-Tory effects (e.g., the Graham-Harrison 2020 election for US senator from SC).

  • 3
    Probably worth reading fivethirtyeight.com/features/… Observation "The reason there’s no long-running polling bias is because pollsters try to correct for their mistakes. That means there’s always the risk of undercorrecting or overcorrecting."
    – James K
    Nov 14 '20 at 8:39
  • 2
    Can you write this less confusingly? Nov 15 '20 at 17:15

538 does a good job of analysing polls:

1972    R+24            R+23    -1
1976    D+1             D+2     -1
1980    R+2             R+10    +8
1984    R+18            R+18    0
1988    R+10            R+8     -2
1992    D+7             D+6     +1
1996    D+13            D+9     +4
2000    R+4             D+1     -5
2004    R+2             R+2     0
2008    D+7             D+7     0
2012    D+0             D+4     -4
2016    D+4             D+2     +2
2020    D+8             D+4*    +4

Pollsters do attempt to correct for their previous errors which is why there isn't a general trend of favouring Republicans or Democrats. The linked article also notes that attempts to correct for "shy tories" in 2017 resulted in a polling error the other way.

Pollsters try to weight their samples so that bias and errors are reduced. I see no long term or clear bias in the above numbers.

  • 1
    538’s Nate Silver also noted this week that we don’t have final votes for 2020 yet, so that 4 point gap may change. Nov 14 '20 at 15:33
  • 1
    Its currently closer to 4.5, the 4 point gap assumes it will tighten slightly as the final votes are counted.
    – James K
    Nov 14 '20 at 15:59

Not the answer you're looking for? Browse other questions tagged .