8

enter image description here

On November 3rd, Arnon Mishkin, a statistician and director of the Fox News decision team, called Arizona for Biden with ~25% of votes still uncounted. When called upon to defend his decision, he explained on air to the Fox News desk that Trump would not be able to recover Biden's 7-point lead. He says,

It has been in the category that we call 'knowable but not callable' for about an hour. Um, we finally called it right now, ... [...]

and a little later adds,

This is a call that's sitting in our statistical models at sort of, at a t-stat of 4 or more in all the different ways [...] That means we're 4 standard deviations from being wrong [...]

I forgot about it since, but coming back to check out Arizona's results, I was pretty surprised to see what a comeback Trump did make (courtesy of New York Times' election tracker):

enter image description here

So I'm not sure if...

  1. I'm misinterpreting the statement or statistics, i.e. it's not surprising at all, even mostly expected that this is how it would go, or

  2. some unprecedented factor came into view regarding the last ~25% of uncounted votes (at the time of the call), or

  3. this truly was a freak occurrence: the 0.01% tail of universes actually happened during one of the most controversial elections of modern day, and Trump regained nearly all of Biden's 7-point lead, which again Mishkin was not only so confident as to call it for Arizona with ~25% of votes still uncounted, but before any other network and on Fox at that.

I'm not at all disputing the election results by the way. It's because I am following experts that I want to know what I'm not getting.


This is an edited question, so post-emptively responding to the "doesn't matter, it's within range" line of argument: Yes, I do understand that Mishkin wasn't saying it was impossible. But if I were on the test flight of a wholly new aircraft and it nearly crashed, but the pilot tells me it was "within range" of their models, I'd still ask, "Hey um, guys? Is there something you wanna tell me?" Because at that point you have to wonder, what's more likely? On the virgin voyage of this totally new thing, you just had a stroke of bad luck—about 4 standard deviations off-center? Or that something unexpected happened? (I'm appealing to a Bayesian mode of thought here.) (Also, in an interview the next day, Mishkin tells a story of how they came up with a brand new model using phone calls and Internet surveys, particular targeting early voters, to try to get a better estimation regarding mail-in ballots.)

6
  • 3
    The question of whether they are surprising is subjective. Better to ask: do they fall within the margin of error of the prediction? – Obie 2.0 Nov 18 '20 at 21:52
  • I guess I'm really asking why no one seems surprised. Like maybe I misunderstand, or some other circumstance appeared, etc. – Andrew Cheong Nov 18 '20 at 21:58
  • The question to ask is whether Mishkin's model (whatever it was) accounted for actual vote vs expected vote or was just based on trends/results from the counting to that point. Because vote counting doesn't follow statistical patterns. The final number doesn't change, we just slowly reveal it in arbitrary increments. – Bobson Nov 19 '20 at 0:06
  • 1
    I don't think that Fox'd Decision Desk has made enough details public about their model to answer a q like this. – Fizz Nov 19 '20 at 0:26
  • @Bobson - Hm. Maybe I don't understand what you mean. In an interview he said that the model was a new one that surveyed early voters via phone and Internet to get a better estimation of what mail-in ballots would look like. But also, he's a pretty well-respected statistician and a very precise type of speaker; I don't think he would use statistical terms for something that doesn't follow a model of statistics. – Andrew Cheong Nov 19 '20 at 7:21
4

An important factor is that later counted ballots in Arizona favored Trump, a pattern that was the opposite of what was seen in many other competitive states. This pattern was unusual, but it was not unexpected. However, somebody who was not aware of the different conditions in Arizona may have been surprised.

There was a lot of discussion about Fox's call and the subsequent narrowing of the count in FiveThirtyEight's Live Blog (where the following two quotes were posted).

Sarah Frostenson, Nov. 4, 9:09 pm:

But let’s look at Arizona. Biden currently leads here by 3.4 points, but there are around 600,000 ballots left to be counted statewide, according to ABC News. About 400,000 of those are in Maricopa County, 250,000 of which were cast this weekend. That’s notable because while mail ballots often favor Democrats, that’s not the case in Arizona, which has a long history of voting by mail. Trump has already eaten into Biden’s lead there in the last 24 hours, although two outlets, Fox News and the AP, have called the race for Biden (ABC News has not yet issued a projection).

It's hard to answer this question fully without knowing Mishkin's team's methodology. We can see from looking at other groups' discussion and predictions at the time that the race was expected to be close. Mishkin's model may have given a t-stat of greater than four, but the model may have been flawed. It doesn't appear to have been the consensus that the outcome was clear—many outlets did not call the race so soon, and some experts expressed doubt as to whether the call should have been made.

Nate Silver Nov. 4, 9:33 pm:

I don’t know, I guess I’d say that Biden will win Arizona if you forced me to pick, but I sure as heck don’t think the state should have been called by anyone, and I think the calls that were previously made should be retracted now.

Nate Cohn 11:54 PM Nov 3, 2020 (Twitter)

I certainly agree that Biden's favored in Arizona, but look the Fox call there is too fast for me. It reminds me of when they called Wisconsin for Trump early in '16, when he was up like 4 points at 70% counted. It worked out! But that was not a 99.99% certainty situation IMO

1

Prediction markets were signalling the race in Arizona was quite a bit far from being over when Mishkin and his team made this call, so there's good reason to believe that the outcome was not as certain as Mishkin's "t-stat of 4" may lead one to believe.

The challenge in this kind of forecasting is to account for the sampling bias involved in counting votes. If you were certain that there is no sampling bias, counting something like 1600 votes would already give you a four sigma certainty that Biden is going to win the election if the results in your sample give Biden %55 of the vote. However, we know that this is very far from being true in general: almost all elections exhibit some sampling bias in votes that are counted early versus those that are counted late. The task of the forecaster is then to apply a correction for this sampling bias.

There are various different approaches to this, and in this domain the most popular might be poststratification: if you know that %20 of the vote is going to come from rural areas but in your sample only %5 of it comes from rural areas, you can take a weighted average to quadruple the weight of the rural vote while estimating the final vote share. This technique also comes with standard error estimates and can be applied with arbitrarily many correction variables. Of course, you never know whether your poststratification model is actually a good model or not, and that makes the reporting of t-stats divorced from any context quite meaningless. While in a single model it's extremely unlikely for a four-sigma event to happen under the null hypothesis, it's quite easy for only a minor change in the specification of the model to change a four-sigma outcome into a one-sigma outcome, especially when the result of the election is expected to be this close.

I would strongly discourage people from taking "four sigma" results "too literally" when the underlying model which generates the t-statistics is not known to be reliable. It's perfectly legitimate to put faith in t-statistics coming out of quantum mechanics; it's not so legitimate to put faith in t-statistics coming out of poststratification models that aren't known to be well established. I think the Arizona call by Fox News was simply a mistake given the data they had available to them at the time, and there was a good chance they would turn out to be wrong a priori.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .