18

Here are the current U.S. presidential election forecasts from Predictwise and FiveThirtyEight.

                  Clinton    Trump
Predictwise          67        33
FiveThirtyEight      52        47
Math details:

There is a 29 point gap between the two forecasts ((67-33)-(52-47) = 29). This is a factor of 6 times the difference in the closer estimate (29 / (52-47) = 6).

Bottom line:

Although both forecasts favor the same candidate, the results are wildly different in magnitude.

Questions:
  • What explains the large difference between the two forecasts?
  • Given that FiveThirtyEight correctly predicted the results of every state in the most recent election, which forecast makes a more compelling case for accuracy?
  • 20
    We can let you known which has the most compelling case for accuracy on November 9th, if you still want to know then. – Jimmy M. Jul 27 '16 at 15:25
  • 5
    I am pretty sure their model types differ significantly, so comparing them may not be a meaningful thing to do (are you looking and 538 nowcast? polls-only? polls-plus?) – user4012 Jul 27 '16 at 15:26
  • 1
    I think you'd get a very different, very interesting perspective if you ask on CrossValidated (Stats.SE). – Max Jul 27 '16 at 21:26
  • 4
    A man with a watch knows what time it is, a man with two is never quite sure ... – DarkHeart Jul 28 '16 at 6:24
  • 1
    Looks like neither – K Dog Dec 3 '16 at 9:56
24

They're both "correct", in so far as predictions can be. They're measuring different things.

FiveThirtyEight has three models, with the default one at that site being the "polls-only" one. They go into their methodology here, and summarize it:

Step 1: Collect, weight and average polls.
Step 2: Adjust polls.
Step 3: Combine polls with demographic and (in the case of polls-plus) economic data.
Step 4: Account for uncertainty and simulate the election thousands of times.

In other words, it's purely data-driven: Polls + demographics = prediction.

Predictwise, on the other hand, aggregates "Prediction Markets (Betfair, PredictIt, Hypermind), Polling (HuffPost Pollster), [and] Bookie (OddsChecker).". Their method is described as

Step 1: construct prices from the back/sell, lay/bid, and last transaction odd/price in the order book. We always take the average of the highest price traders are willing to buy a marginal share and the lowest price people are willing to sell a marginal share, unless the differential is too large or does not exist.
Step 2: correct for historical bias and increased uncertainty in constructed prices near $0 or $1. We raise all of the constructed prices to a pre-set value depending on the domain.
Step 3: normalized to equal 100% for any mutually exclusive set of outcomes.

In other words, it's mostly market-driven: People's bets -> Market -> Prediction. (In theory, there is polling data used in there, but I can't find any details on that.)


TL;DR

FiveThirtyEight is predicting how people will vote based on polls and demographics. Predictwise is aggregating who people think will win based on market data. Two different methodologies to produce two different statistics, resulting in two different percentages.

  • 3
    No! That's not true at all. Study fivethirtyeight.com more closely. They are making a predictive forecast on the final election result. Not the popular vote split or any other statistic. (That is precisely the reason why I am comparing fivethirtyeight and not some random poll.) This is an apples-to-apples comparison. – Rain Willow Jul 27 '16 at 16:40
  • The fact Predictwise is aggregating people's beliefs about who will win seems to make it one step removed from the question of who will win. Predictwise uses my belief about the winner, which is based on polling numbers and other predictions of how people will vote. Isn't it the prediction of how people will vote what we're really interested in, though? – Nuclear Wang Jul 27 '16 at 18:10
  • 1
    @Mowzer 538 are making a predictive forecast of the election result based on polls and demographics.538 tends to be more sensitive to state polls rather than nationwide, but The answer by bobson is correct. Different methodologies -> different results. Currently polls are indicating a trump marginal lead in ohio, and florida. – James K Jul 27 '16 at 22:07
  • 2
    @Mowzer - I don't think I said that they're modeling the popular vote. They model the vote shares of each state and determine what that means for the electoral college. That results in them predicting who gets the most electoral votes. – Bobson Jul 28 '16 at 3:14
  • @Matt - Depends what you're looking for. It's the difference between asking someone "Who do you want to win?" and "Who do you think will win?" Both are valid questions, but the answers may be very different, even from the same person, and they aren't necessarily correlated. – Bobson Jul 28 '16 at 3:23
3

You're looking at the 538 "Polls Only" forecast, which is not going to be as useful for this sort of comparison. The "Polls Plus" forecast includes historical data and other details which makes for a better prediction. Note that when you switch to that, you get (currently) 61% Clinton-39% Trump, which is much closer to the Predictwise estimate. Predictwise will likely fall closer to that than to the polls-only forecast, in particular because it is similarly taking into account additional information beyond simply the polls (via the prediction markets).

The bigger question - is Predictwise's model (prediction markets) better than FiveThirtyEight's model (polling, certain historical trends, etc.) - is an open question, and likely one which will vary over time as both FiveThirtyEight and Predictwise incorporate additional information into their models. Prediction markets also add an additional layer of complication into that comparison as the bettors in the prediction markets know the FiveThirtyEight predictions, so this isn't a comparison of two independent predictors.

  • Didn't notice your answer until after I also noticed the same thing and updated mine. However, since you're correct, +1 from me. – T.E.D. Jul 28 '16 at 21:48
  • Whether "Polls only" or "Polls plus" is going to be a better prediction is uncertain. That's why they offer both. – Bobson Jul 29 '16 at 4:16
  • @Bobson - As I mentioned in my answer, even the folks at 538 are pretty sure that their "Polls plus" is the better model to look at for the next week or so. – T.E.D. Jul 29 '16 at 13:32
  • @Bobson Which is better is entirely open question. Which is more appropriately compared to Predictwise, though, I think is not. – Joe Jul 29 '16 at 14:12
  • @Joe - I don't think either one is a particularly good comparison to Predictwise. As I say in my answer, they're measuring two different things. – Bobson Jul 29 '16 at 14:23
1

Neither.

Now that the national conventions are over the actual campaigns have just begun. It's still a long time until the election. A lot can and will happen in the next three months which will have a major impact on the results.

  • I believe Nate Silver has written extensively on how accurate of a predictor polls are at various points in time. Saying "neither" doesn't cut it. – djechlin Jul 28 '16 at 21:25
  • I feel like you missed the point of the question. Neither forecast is saying "Hillary will become president." They're just both saying "She is more likely to become president." That uncertainty is expressly because of how long it is until the election and how much can happen. This answer doesn't explain why the two models differ so much on the odds they're giving, though. – Bobson Jul 29 '16 at 4:19
1

Given that FiveThirtyEight correctly predicted the results of every state in the most recent election, which forecast makes a more compelling case for accuracy?

But it did that at the end of the election, with polls from the final month. Neither has access to that information now. As such, expecting either to successfully predict the election is problematic.

That said, I would pick the 538 prediction over Predictwise. The Predictwise method just failed, badly, in the Brexit vote. The biggest problem is that it is overly certain. At this point in the election, neither candidate has a 70% chance of winning. It's a much closer election than that, and too many things could happen in the interim.

What if there's a smoking gun in the 30,000 deleted emails? What if Congress passes a law releasing Trump's tax returns and there's a smoking gun there? Or a new Deep Throat reveals a smoking gun in Trump's tax returns or business dealings. (A smoking gun in this context means conclusive evidence, like someone holding a smoking gun next to a dead body.)

Election polls and predictions have historically shifted by significant amounts from convention time to the final election.

1

You should believe both models in some combination.

Different models like the ones Predictwise and 538 use will naturally predict different results. Given that you or I don't really have evidence* to claim that one model is unconditionally better than the other, when forming your own idea of the probability of Clinton or Trump winning, an intuitive approach would simply be to average the predictions made by the models. This is similar to how Bayesian statistics interprets and applies models.

*Alternatively, if you did have some evidence, you'd give different weights to each model in the average - perhaps, 60 percent to 538 and 40 percent to Predictwise or whatnot. Determining what those weights would be though, is an entirely different ballgame and is not likely to have easy answers.

1

You've asked three separate questions, and I think some have been answered but I think the main question warrants a much broader answer.

Which to believe: FiveThirtyEight or Predictwise?

I believe the foundation of this question is determining which of the two options presented is "better" or optimal when compared with the other. Both of these forecasts provide probabilistic forecasts. Both consider polling data as inputs. The big differences are inputs that have less significant weight when compared to poll data, but can still have remarkable effects on the output. To avoid writing a novel, I'll try to summarize what I think are key differences.

538 pays a lot of attention to the quality, decay, and noise from polling data. Silver uses Pollster rankings, trend line adjustments, and poll sample adjustments to enhance the performance of a run-of-the-mill polling average.

Predictwise includes inputs from emerging datasets like prediction markets and social media panels to fine tune it's outputs.

To directly answer your question--you should pay attention to both prediction outputs. They utilize different approaches, both with unique weaknesses and strengths. As with most predictions the wisdom of the crowd provides you with significantly more accurate results than utilizing a singular source.

What explains the large difference between the two forecasts?

Inputs and methodology. You can read about 538's here and PredictWise here

Given that FiveThirtyEight correctly predicted the results of every state in the most recent election, which forecast makes a more compelling case for accuracy?

This question can only be answered by determining how you evaluate models. Nate Silver has a thorough write up here with quotes from Rothschild from PredictWise.

1

After a bit more research, I believe your problem is that you're looking at the wrong 538 model. 538 has 2 models it uses; one that's pure polls, and one that tries to take account of 2nd-order effects, which they call "polls plus".

We are currently at a point where polls will be seeing the RNC "bounce", and will not yet be seeing the (theoretically upcoming) DNC "bounce". This may be causing the two to artificially diverge. 538's "polls plus" model tries to account for the bounce. And of course savvy betters know about the convention bounces, so they will be accounting for it as well.

This is why even 538's own political editor is advising to ignore their polls model for a while: enter image description here

Their current polls-plus model has the split at 61.6% to 38.3%. That leaves the, er, delta between their point spread and Predictwise's as only about 10 points.

Even that may seem a lot, but again this time between the conventions is typically the most volatile of the entire campaign for polls. Wait a couple of weeks for the bounces to die down, and we should have a better picture.


As a historical note: There was in fact a large DNC bounce, a decay back toward a close race, another large bounce, and another decay. In the final analysis, 538 gave Trump a larger chance to win (about 30%) than the betting models (about 10% for Predictwise), and he did in fact win. So it looks like between the two of them you would have been better off paying attention to 538 (and some basic prob-stat).

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .