Why do election predictions by FiveThirtyEight and Predictwise differ so much?

Question

Here are the current U.S. presidential election forecasts from Predictwise and FiveThirtyEight.

                  Clinton    Trump
Predictwise          67        33
FiveThirtyEight      52        47

Math details:

There is a 29 point gap between the two forecasts ((67-33)-(52-47) = 29). This is a factor of 6 times the difference in the closer estimate (29 / (52-47) = 6).

Bottom line:

Although both forecasts favor the same candidate, the results are wildly different in magnitude.

Questions:

• What explains the large difference between the two forecasts?
• Given that FiveThirtyEight correctly predicted the results of every state in the most recent election, which forecast makes a more compelling case for accuracy?

Answer 1

They're both "correct", in so far as predictions can be. They're measuring different things.

FiveThirtyEight has three models, with the default one at that site being the "polls-only" one. They go into their methodology here, and summarize it:

Step 1: Collect, weight and average polls.
Step 2: Adjust polls.
Step 3: Combine polls with demographic and (in the case of polls-plus) economic data.
Step 4: Account for uncertainty and simulate the election thousands of times.

In other words, it's purely data-driven: Polls + demographics = prediction.

Predictwise, on the other hand, aggregates "Prediction Markets (Betfair, PredictIt, Hypermind), Polling (HuffPost Pollster), [and] Bookie (OddsChecker).". Their method is described as

Step 1: construct prices from the back/sell, lay/bid, and last transaction odd/price in the order book. We always take the average of the highest price traders are willing to buy a marginal share and the lowest price people are willing to sell a marginal share, unless the differential is too large or does not exist.
Step 2: correct for historical bias and increased uncertainty in constructed prices near $0 or $1. We raise all of the constructed prices to a pre-set value depending on the domain.
Step 3: normalized to equal 100% for any mutually exclusive set of outcomes.

In other words, it's mostly market-driven: People's bets -> Market -> Prediction. _{(In theory, there is polling data used in there, but I can't find any details on that.)}

---

TL;DR

FiveThirtyEight is predicting how people will vote based on polls and demographics. Predictwise is aggregating who people think will win based on market data. Two different methodologies to produce two different statistics, resulting in two different percentages.

Answer 2

You're looking at the 538 "Polls Only" forecast, which is not going to be as useful for this sort of comparison. The "Polls Plus" forecast includes historical data and other details which makes for a better prediction. Note that when you switch to that, you get (currently) 61% Clinton-39% Trump, which is much closer to the Predictwise estimate. Predictwise will likely fall closer to that than to the polls-only forecast, in particular because it is similarly taking into account additional information beyond simply the polls (via the prediction markets).

The bigger question - is Predictwise's model (prediction markets) better than FiveThirtyEight's model (polling, certain historical trends, etc.) - is an open question, and likely one which will vary over time as both FiveThirtyEight and Predictwise incorporate additional information into their models. Prediction markets also add an additional layer of complication into that comparison as the bettors in the prediction markets know the FiveThirtyEight predictions, so this isn't a comparison of two independent predictors.

Answer 3

Neither.

Now that the national conventions are over the actual campaigns have just begun. It's still a long time until the election. A lot can and will happen in the next three months which will have a major impact on the results.

Answer 4

Given that FiveThirtyEight correctly predicted the results of every state in the most recent election, which forecast makes a more compelling case for accuracy?

But it did that at the end of the election, with polls from the final month. Neither has access to that information now. As such, expecting either to successfully predict the election is problematic.

That said, I would pick the 538 prediction over Predictwise. The Predictwise method just failed, badly, in the Brexit vote. The biggest problem is that it is overly certain. At this point in the election, neither candidate has a 70% chance of winning. It's a much closer election than that, and too many things could happen in the interim.

What if there's a smoking gun in the 30,000 deleted emails? What if Congress passes a law releasing Trump's tax returns and there's a smoking gun there? Or a new Deep Throat reveals a smoking gun in Trump's tax returns or business dealings. (A smoking gun in this context means conclusive evidence, like someone holding a smoking gun next to a dead body.)

Election polls and predictions have historically shifted by significant amounts from convention time to the final election.

Answer 5

You should believe both models in some combination.

Different models like the ones Predictwise and 538 use will naturally predict different results. Given that you or I don't really have evidence* to claim that one model is unconditionally better than the other, when forming your own idea of the probability of Clinton or Trump winning, an intuitive approach would simply be to average the predictions made by the models. This is similar to how Bayesian statistics interprets and applies models.

*Alternatively, if you did have some evidence, you'd give different weights to each model in the average - perhaps, 60 percent to 538 and 40 percent to Predictwise or whatnot. Determining what those weights would be though, is an entirely different ballgame and is not likely to have easy answers.

Answer 6

You've asked three separate questions, and I think some have been answered but I think the main question warrants a much broader answer.

Which to believe: FiveThirtyEight or Predictwise?

I believe the foundation of this question is determining which of the two options presented is "better" or optimal when compared with the other. Both of these forecasts provide probabilistic forecasts. Both consider polling data as inputs. The big differences are inputs that have less significant weight when compared to poll data, but can still have remarkable effects on the output. To avoid writing a novel, I'll try to summarize what I think are key differences.

538 pays a lot of attention to the quality, decay, and noise from polling data. Silver uses Pollster rankings, trend line adjustments, and poll sample adjustments to enhance the performance of a run-of-the-mill polling average.

Predictwise includes inputs from emerging datasets like prediction markets and social media panels to fine tune it's outputs.

To directly answer your question--you should pay attention to both prediction outputs. They utilize different approaches, both with unique weaknesses and strengths. As with most predictions the wisdom of the crowd provides you with significantly more accurate results than utilizing a singular source.

What explains the large difference between the two forecasts?

Inputs and methodology. You can read about 538's here and PredictWise here

Given that FiveThirtyEight correctly predicted the results of every state in the most recent election, which forecast makes a more compelling case for accuracy?

This question can only be answered by determining how you evaluate models. Nate Silver has a thorough write up here with quotes from Rothschild from PredictWise.

Answer 7

After a bit more research, I believe your problem is that you're looking at the wrong 538 model. 538 has 2 models it uses; one that's pure polls, and one that tries to take account of 2nd-order effects, which they call "polls plus".

We are currently at a point where polls will be seeing the RNC "bounce", and will not yet be seeing the (theoretically upcoming) DNC "bounce". This may be causing the two to artificially diverge. 538's "polls plus" model tries to account for the bounce. And of course savvy betters know about the convention bounces, so they will be accounting for it as well.

This is why even 538's own political editor is advising to ignore their polls model for a while:

Their current polls-plus model has the split at 61.6% to 38.3%. That leaves the, er, delta between their point spread and Predictwise's as only about 10 points.

Even that may seem a lot, but again this time between the conventions is typically the most volatile of the entire campaign for polls. Wait a couple of weeks for the bounces to die down, and we should have a better picture.

---

As a historical note: There was in fact a large DNC bounce, a decay back toward a close race, another large bounce, and another decay. In the final analysis, 538 gave Trump a larger chance to win (about 30%) than the betting models (about 10% for Predictwise), and he did in fact win. So it looks like between the two of them you would have been better off paying attention to 538 (and some basic prob-stat).