How did Arizona get so close to upsetting Arnon Mishkin's "t-stat of 4" statement defending his decision team's call?

Question

On November 3rd, 2020, Arnon Mishkin, a statistician and director of the Fox News decision team, called Arizona for Biden with ~25% of votes still uncounted. When called upon to defend his decision, he explained on air to the Fox News desk that Trump would not be able to recover Biden's 7-point lead. He says,

It has been in the category that we call 'knowable but not callable' for about an hour. Um, we finally called it right now, ... [...]

and a little later adds,

This is a call that's sitting in our statistical models at sort of, at a t-stat of 4 or more in all the different ways [...] That means we're 4 standard deviations from being wrong [...]

I forgot about it since, but coming back to check out Arizona's results, I was pretty surprised to see what a comeback Trump did make (courtesy of New York Times' election tracker):

So I'm not sure if...

1. I'm misinterpreting the statement or statistics, i.e. it's not surprising at all, even mostly expected that this is how it would go, or

2. some unprecedented factor came into view regarding the last ~25% of uncounted votes (at the time of the call), or

3. this truly was a freak occurrence: the 0.01% tail of universes actually happened during one of the most controversial elections of modern day, and Trump regained nearly all of Biden's 7-point lead, which again Mishkin was not only so confident as to call it for Arizona with ~25% of votes still uncounted, but before any other network and on Fox at that.

I'm not at all disputing the election results by the way. It's because I am following experts that I want to know what I'm not getting.

---

^{This is an edited question, so post-emptively responding to the "doesn't matter, it's within range" line of argument: Yes, I do understand that Mishkin wasn't saying it was impossible. But if I were on the test flight of a wholly new aircraft and it nearly crashed, but the pilot tells me it was "within range" of their models, I'd still ask, "Hey um, guys? Is there something you wanna tell me?" Because at that point you have to wonder, what's more likely? On the virgin voyage of this totally new thing, you just had a stroke of bad luck—about 4 standard deviations off-center? Or that something unexpected happened? (I'm appealing to a Bayesian mode of thought here.) (Also, in an interview the next day, Mishkin tells a story of how they came up with a brand new model using phone calls and Internet surveys, particular targeting early voters, to try to get a better estimation regarding mail-in ballots.)}

Answer 1

An important factor is that later counted ballots in Arizona favored Trump, a pattern that was the opposite of what was seen in many other competitive states. This pattern was unusual, but it was not unexpected. However, somebody who was not aware of the different conditions in Arizona may have been surprised.

There was a lot of discussion about Fox's call and the subsequent narrowing of the count in FiveThirtyEight's Live Blog (where the following two quotes were posted).

Sarah Frostenson, Nov. 4, 9:09 pm:

But let’s look at Arizona. Biden currently leads here by 3.4 points, but there are around 600,000 ballots left to be counted statewide, according to ABC News. About 400,000 of those are in Maricopa County, 250,000 of which were cast this weekend. That’s notable because while mail ballots often favor Democrats, that’s not the case in Arizona, which has a long history of voting by mail. Trump has already eaten into Biden’s lead there in the last 24 hours, although two outlets, Fox News and the AP, have called the race for Biden (ABC News has not yet issued a projection).

It's hard to answer this question fully without knowing Mishkin's team's methodology. We can see from looking at other groups' discussion and predictions at the time that the race was expected to be close. Mishkin's model may have given a t-stat of greater than four, but the model may have been flawed. It doesn't appear to have been the consensus that the outcome was clear—many outlets did not call the race so soon, and some experts expressed doubt as to whether the call should have been made.

Nate Silver Nov. 4, 9:33 pm:

I don’t know, I guess I’d say that Biden will win Arizona if you forced me to pick, but I sure as heck don’t think the state should have been called by anyone, and I think the calls that were previously made should be retracted now.

Nate Cohn 11:54 PM Nov 3, 2020 (Twitter)

I certainly agree that Biden's favored in Arizona, but look the Fox call there is too fast for me. It reminds me of when they called Wisconsin for Trump early in '16, when he was up like 4 points at 70% counted. It worked out! But that was not a 99.99% certainty situation IMO

Answer 2

I think there's a fundamental misunderstanding of statistical models here. When political pollsters do a statistical analysis, they are testing whether the number of voters in the full population is greater for Candidate A or Candidate B, based on data they get from polling samples and certain theoretical presumptions built into their model. Nothing in the statistical analysis indicates the distance between the candidates; it only indicates the statistician's confidence that one candidate is winning.

A 4-sigma result means that the statistician is more than 99.4% confident that Candidate A has a greater number of citizens voting for him/her. It doesn't indicate whether the Candidate A has 50,000 more votes, or 5000 more votes, or 500, or 5. It merely says that there is less than a 0.6% chance that Candidate B actually has more votes. Statistical results are a measure of significance, not a measure of size, and there's no sense reading more into it.

Answer 3

Prediction markets were signalling the race in Arizona was quite a bit far from being over when Mishkin and his team made this call, so there's good reason to believe that the outcome was not as certain as Mishkin's "t-stat of 4" may lead one to believe.

The challenge in this kind of forecasting is to account for the sampling bias involved in counting votes. If you were certain that there is no sampling bias, counting something like 1600 votes would already give you a four sigma certainty that Biden is going to win the election if the results in your sample give Biden %55 of the vote. However, we know that this is very far from being true in general: almost all elections exhibit some sampling bias in votes that are counted early versus those that are counted late. The task of the forecaster is then to apply a correction for this sampling bias.

There are various different approaches to this, and in this domain the most popular might be poststratification: if you know that %20 of the vote is going to come from rural areas but in your sample only %5 of it comes from rural areas, you can take a weighted average to quadruple the weight of the rural vote while estimating the final vote share. This technique also comes with standard error estimates and can be applied with arbitrarily many correction variables. Of course, you never know whether your poststratification model is actually a good model or not, and that makes the reporting of t-stats divorced from any context quite meaningless. While in a single model it's extremely unlikely for a four-sigma event to happen under the null hypothesis, it's quite easy for only a minor change in the specification of the model to change a four-sigma outcome into a one-sigma outcome, especially when the result of the election is expected to be this close.

I would strongly discourage people from taking "four sigma" results "too literally" when the underlying model which generates the t-statistics is not known to be reliable. It's perfectly legitimate to put faith in t-statistics coming out of quantum mechanics; it's not so legitimate to put faith in t-statistics coming out of poststratification models that aren't known to be well established. I think the Arizona call by Fox News was simply a mistake given the data they had available to them at the time, and there was a good chance they would turn out to be wrong a priori.