Statistical Primer- The Infamous M O E

[DemocratSinceBirth]

Just because a candidate's lead is inside the Margin Of Error doesn't mean he or she isn't really ahead, ergo:

Just because the poll results are within the poll's Margin of Error (or MoE)....

... does not make it a statistical dead heat - if you think it does, you have been listening to too many talking heads on TV.

...

First of all the the MoE works both ways. With a 4.0 MoE the results could be Sanders +1 (least likely), A tie, Clinton +1, Clinton +2, Clinton +3 (most likely), Clinton +4, Clinton +5, Clinton +6, Clinton +7 (least likely). Note that with most of the possible outcomes, including the most likely outcome (Clinton +3), Clinton wins.

In addition, as you move one step at a time away from the most likely outcome (Clinton +3) towards the edges (Sanders +1 and Clinton +7) the chances of those results happening get less and less. The least likely outcomes by far are Sanders +1 and Clinton +7.

If you want to get technical, there is a 95% chance with this poll that the result will be within the MoE of 4.0, and a 5% chance that it will be off more than 4.0%. Since the graph of this poll's possible results is the area under a bell shaped curve, there is actually just a 2.5% chance that Sanders could win by more than +1 and 2.5% that Clinton could win by more than +7, so such results are highly unlikely.

-Cajun Blazer

[NurseJackie]

1. Excellent post!

[DemocratSinceBirth]

4. Thank Cajun Blazer. He explained it.

I did know the MOE covers a theoretical range of outcomes but it certainly doesn't mean the candidates are tied.

Last night I aggregated the last five polls and HRC had a lead of 3.3% with a MOE of 1.73%.

Cajun Blazer can do a theoretical range of outcomes with it.

[Jarqui]

2. She used average turnout to project that margin

If the turnout is heavy, usually due to youth and enthusiasm, Bernie probably wins.
If the turnout is light, Hillary will not only win but might hit that +7 you're talking about.

I think she tried to poll enthusiasm. If one looks at average turnout to the campaign events, actions speak louder than words or poll numbers on enthusiasm. Bernie has that - particularly with the youth.

[alcibiades_mystery]

19. I agree to some extent

Iowa will show us more than just who wins Iowa. It will show us the accuracy of the Sanders' camp claims that it can win and actually institute policy due to a ground swell of new voters, young voters, independent voters, non-traditional voters, and the like. If that doesn't materialize in Iowa, then that claim becomes far more dubious, not only for the general election, but for all of the claims about policy change stemming from a transformed, engaged political landscape.

It's a very interesting challenge / promise / problem / opportunity. A big win for Bernie that includes lots of new, enthusiastic participants essentially confirms Sanders' narrative and lends support to his arguments about how he can win the general and govern effectively. A loss resulting from a failure to mobilize a new, enthusiastic participant pool is not only a loss of one state, but a refutation of his entire argument.

[Tom Rinaldo]

3. Of course that is always in the opinion of the pollster at the time

I am not knocking the latest poll - just pointing that out. When polls go wrong for any reason the MOE they initially assumed becomes moot.

[DemocratSinceBirth]

5. I am not disputing the fact that there can be a failure in polling.

Aggregate polling in 00 showed Bush with a slight popular vote lead.

But my point and the point Cajun Blazer made so well is that the MOE represents a theoretical range of outcomes, from the most likely to the least likely, and it certainly doesn't mean that a candidate whose lead is within the Margin Of Error isn't really ahead.

[dsc]

8. No

Yes the polling can be wrong, but it isn't the pollster's opinion (at least if the pollster is not a hack it isn't) but a scientifically designed method to estimate a parameter within a desired range of accuracy.

[DemocratSinceBirth]

10. Precisely

Others are better at explaining the minutiae of polling than I am but the turnout her poll is based on is a function of the percentage of respondents who said they would actually be caucusing.

She didn't guess.

[Paulie]

20. It's based on the model used

With average turnout the number is X. A few percent more turnout and its Y.

One more day and we'll know for sure.

[Tom Rinaldo]

25. "I misspoke!"

I agree that the poll wasn't representing an opinion. That was poor shorthand. A more accurate comment might be "based on that polls design and the weighing of variables. Plus as pollsters always admit, polls are a snapshot in time and things can move fast at the end.

I only note this because there is something about a MOE that is seductive, making this all seem like more of fool proof predictive science than it really is. Exit polls, on the other hand, are a slightly different breed. They are a snapshot of the finish line, within a MOE, if the sampling is representative of the electorate.

[JackRiddler]

6. Nonsense.

The MOE is a fig leaf and this discussion is pollster fantasy, or self-promotion. What matters is methodology, sample size, weighting, definition of "likely," etc., all of which varies greatly between polls (all of which claim similar MOEs as a matter of promotion). And of course what's actually being measured. This isn't a general election or mass-scale consumer behavior, where prediction can work well. Primary polls are already dodgy, but a caucus? It's all about motivated minorities, and the motivation varies right up until the day. Whichever pollster gets the exact numbers of the actual results in a caucus is winning a lottery, but will claim to have divine powers.

[DemocratSinceBirth]

7. I suspect if you polled one thousand Statistics professors

I suspect if you polled one thousand Statistics professors all one thousand of them would say Cajun Blazer's description of the MOE is the technically correct one.

I would literally wager on it because that is what I and every other student who ever took a Statistics course was taught.

Now, a discussion of the vagaries of polling a caucus is a whole different ball of wax.

[NCTraveler]

9. In Iowa, isn't she outside of the MOE in two of the last three? Nt

[DemocratSinceBirth]

11. You would be correct, and...

Last night I aggregated the last five polls and HRC had a lead of 3.3% with a MOE of 1.73%.

Cajun Blazer can do a theoretical range of outcomes with it.

[morningfog]

12. Yes, Hillary is leading slightly in polling, down from 30+ points a months ago.

However, in addition to the normal unknowns (turnout, new voters, weather, etc) it is a caucus and their is a the is candidate who will likely not meet the 15% threshold in most precincts. Even with the reliability of the poll, Bernie is in a GREAT spot.

From down 30 to down 3 with developments between this poll and the caucus.

[Bjorn Against]

13. It is clear that Cajun Blazer does not understand statistics

The margin of error does not just mean that one candidates numbers might be off by 4%, it means that 4% of a candidate's totals could belong to a different candidate. this means that Bernie could actually be +5 and still be within the poll's margin of error. Now this would be an unlikely scenario if the poll was conducted well, but +1 is not an unlikely scenario as Cajun Blazer suggests. All a +1 margin would mean is that the pollster over represented Clinton supporters by 2% and that 2% was actually for Bernie which would put him slightly in the lead.

[DemocratSinceBirth]

14. The margin of error represents a theoretical range of outcomes.

The margin of error represents a theoretical range of outcomes from the most likely to the least likely, all the obscurantism in the world notwithstanding, and the further you get from the reported values the less likely it is to occur, ergo:

However, the estimate a poll gets is in fact the likeliest true value, and the likelihood decreases as we move toward the extreme ends of sampling error.

-Gary Langer

http://blogs.abcnews.com/thenumbers/2007/12/moe-and-mojo.html

Gary Langer is an internationally recognized public opinion researcher with expertise in analysis of political, policy, economic and social attitudes, questionnaire design and survey methodology and management. With more than 25 years in the field, Langer, a longtime director of polling at ABC News, has overseen and analyzed more than 750 surveys on a broad range of topics.

Langer has won two Emmy awards and received nine Emmy nominations – including the first and only to cite public opinion polls – and was honored with the 2010 Policy Impact Award of the American Association for Public Opinion Research for a series of surveys in Afghanistan and Iraq, described in AAPOR’s citation as “a stellar example of high-impact public opinion polling at its finest.” He’s a two-time winner of the University of Iowa-Gallup Award for Excellent Journalism Using Polls, produced a pair of ABC News polls recognized by the Excellence in Media Coverage of Polls Award from the National Council on Public Polls and shared a DuPont-Columbia Award for ABC’s 9/11 coverage. Most recently, Langer and his colleagues shared a David R. Ogilvy Award for Excellence in Advertising Research with ESPN and its partner research firms for their work on fan interest in the College Football Playoffs.

Langer created ABC’s industry-leading poll standards and vetting procedures and has promoted disclosure initiatives through various professional organizations. He’s authored or co-authored nearly 30 scholarly papers and given scores of invited presentations on the meaning and measurement of public attitudes.

http://www.langerresearch.com/our-depth/gary-langer/

[Bjorn Against]

16. Yes, it is a theoretical range of outcomes but Cajun Blazer gets the range wrong

If a poll overrepresents one candidate's numbers those numbers do not generally just disappear, they often go to another candidate which means the actual range of variance is double the margin of error.

Any statistics professor would tell you that Cajun Blazer is wrong in claiming that +1 is the best Bernie could do within the margin of error, +5 would still be in the margin of error.

[DemocratSinceBirth]

18. Maybe, all three of us are saying the same thing differently...

It is my understanding that the MOE applies equally to all values or in this instance candidates.

HRC has 45
SBS has 42

With a MOE of 4 HRC can be as high as 49 and as low as 41

With a MOE of 4 SBS can be as low as 38 and as high as 46


That being said, the reported values are most likely the correct one and the further removed from the reported values the less likely they are to occur. You can reduce the MOE by increasing the sample but landline/cellphone polls are really expensive.

Maybe, all three of us are saying the same thing differently...

Where I suggest the obscurantism comes in is when one calls a lead within the margin of error a tie.

[Bjorn Against]

21. That would be a more accurate statement

The Selzer poll shows a real dead heat especially when you consider that 10% are still undecided. Those 10% could break either way which means the actual results could be outside the margin of error of this poll and the poll would still not necessarily be wrong.

[DemocratSinceBirth]

24. If I were the pollster I would have "forced" the undecideds to choose...

It wouldn't have been a top line number though.

If Clinton loses Selzer loses in the eyes of the media who have made her the oracle of polling, regardless of the fine points of her craft...

There is always a "hot" pollster... Zogby was once a "hot" pollster when he nailed the 96 general almost to the decimal point and 00 when his poll showed Gore winning the pop vote. It wasn't that a Martian couldn't have picked Clinton in 96 it was most polls showed Clinton winning by a larger margin. He won by 8.3%. Most polls showed him with a double digit lead.

[NurseJackie]

15. Kick!

Here's a famous MOE:

[mcar]

17. Really good explanation

I learned something new today.

[DemocratSinceBirth]

26. Thank you

[Depaysement]

22. We'll see

She deeply discounted the percentage of new caucus participants compared to 2008. Those participants break heavily for Sanders. To Selzer's credit, she admits this. It's a huge variable, along with general turnout ratios, that you are completely ignoring

Why do you assume the percentages within the MOE on a bell curve? Plus, there are a whole host of other statistical anomalies that could occur: coverage error, non-response error, weighting error and so on.

[Kilgore]

23. Technically correct, however.........

The average tv viewer could not either comprehend it or take the time to try. The "dead heat" short hand is media's way of handling it.

[Gothmog]

27. Good explanation of margin of error

[Jim Lane]

28. I think this is incorrect as to the DIFFERENCE between two candidates

If there's a 4% MOE in each candidate's share of the vote, what's the MOE for the gap between them? I'm pretty sure it's greater than 4%.

For example, let's say the poll result is Clinton 45%, Sanders 42%. It's correct to say that there's no more than a 5% chance that Clinton is outside the band 41% to 49%.

But she doesn't have to be outside that band for Sanders to win. If each number is off by just 1.5% (in the direction favorable to Sanders), then the two candidates actually do tie even with neither number being off by anywhere close to 4%. Any difference slightly more favorable to Sanders means that he wins.

The result is that, if you want to continue to use MOE to refer to the 95% confidence interval, then the MOE for the gap between the two numbers is greater than 4%. For a Sanders win, it's not necessary that actual results exhibit either of the unlikely events (i.e., an individual result that's outside the eight-point band of the MOE).

On the other hand, you can't just double the MOE of an individual number to get the MOE for the gap. That would mean that the poll result for the gap was Clinton by 3% plus or minus 8%. But "minus 8%" would mean Sanders winning by 5% instead of losing by 3%. That would happen only if both candidate totals fell outside the 95% confidence interval. If the chance that Clinton will be outside the range of 41 to 49 is 5%, then the chance that she's outside her range AND that Sanders is outside his range (of 38 to 46) is obviously less than 5%. Therefore, you can narrow the stated MOE (make it less than 8%) and still be within the 95% confidence interval.

All this applies to calculating the MOE for the difference between two numbers in two different polls. For example, if two polls with the same MOE of 4% give Sanders 42% in Iowa and 53% in New Hampshire, what's the MOE for the margin by which Sanders in New Hampshire is outperforming Sanders in Iowa? I think it's more than 4% but less than 8%. If instead you ask the typically more interesting question of comparing two candidates' results in the same poll, the calculation would probably be different because there's a covariance between the two numbers.

[DemocratSinceBirth]

29. Please see Post 18.

[Jim Lane]

31. OK, I've re-read Post 18 and it doesn't address my point

I agree with you that the MOE of 4% applies to the Clinton number and to the Sanders number. For the reasons I stated, though, I don't think it applies to the (Clinton minus Sanders) number.

[Recursion]

30. And even the MoE is itself predicated on the turnout priors being correct

That is, if the polling firm's model of voter turnout is correct, then it is 95% likely that the outcome will be within the MoE of the poll. There's actually two elements of uncertainty here, and only one can be quantified.