Understanding margins of error

[Silent3]

I was listening to Lawrence O'Donnell earlier this week discussing some polls, and talking about what the polls said when you take into account their margins of error.

What he said was roughly correct, in a non-mathy sort of way, but also off enough to be a bit misleading.

First of all, margins of errors don't say anything about how good a polling model is, so it's not an expression of faith in the model. What the margin of error tells you is, if the model is good, how much would random variation in the particular people who get sampled typically throw off how well the model reflects reality.

Also, if a poll says, say, Biden is likely to get 52% of the vote, with a margin of error of ±3%, that does NOT mean Biden is just as likely to get 49% as he is to get 52% -- it's a "bell shaped curve" -- and values near the middle are favored over values near the edges of the given range.



Margins of error are typically stated for a 95% confidence interval. So this example 52±3% will be in the range 49-55 95% of the time. 5% of the time the real result could even be higher or lower than that ±3%. Only 2.5% of the time would Biden be at 49% or lower, Only 2.5% of the time would Biden be at 55% or higher. Typical results will cluster more towards the middle of the range.

Say that Trump is polling at 48% in the same poll. As O'Donnell explained it, he correctly pointed out that you have to apply the margin of error to both numbers, so even though 48% and 52% are 4% apart, the results overlap, with Trump possibly going as high as 51%, and Biden possibly going as low as 49%.

What O'Donnell got wrong, however, is speaking about this situation as if, "Hey, this essentially is a tie", since the margins of error overlap. Nope! This would still be a poll that looks much better for Biden than it does for Trump.

Without getting into the exact math, for Trump to actually be ahead in a poll like that requires that Trump pushes well into the more unlikely upper end of his range at the same time Biden happens to fall into the more unlikely lower end of his range. The sampling errors aren't very likely, however, to line up in just that way very often.

[Wounded Bear]

1. K & R...for visibility...nt

[PCIntern]

2. You explained much better than I could

Which is why I never attempted a post like yours. K and r.

[DonaldsRump]

3. Am so glad there are folks like you on DU

Statistics was not something I excelled at school...and I run away from it today.

K&R!

[jimfields33]

6. You are not alone

I got a D in statistics and the school only accepted C- and above. Took it again next semester with same professor and got a C-. I’m convinced he passed me just to get rid of me. I think they let a few lost causes get a participation grade. Lol.

[BComplex]

4. Thanks Silent3! That really helps me to understand the polls.

Good visuals, too.

[Spazito]

5. Thank you very much, very informative! n/t

[beachbumbob]

7. to me the only use of polls is to indicate the trends and why I was skeptical in 2016 as everything

showed Hillary was dropping the moment Comey kneecapped her campaign. Nothing of the like is indicated in 2020, the exact opposite

[Silent3]

9. The trends were a factor, and so were modeling errors

With both candidates having such high negatives (Clinton very unfairly so), the likely voter models were probably shot to hell.

Even so, I think 538 still had Trump at about a 30% chance of winning, so leaning toward Clinton, but far from a sure thing. I was very worried at the time, because so close to a 1-in-3 chance of Trump winning was terrifying to me.

And as it turned out, not just from the way the election went, but how Trump has behaved as President, I was right to be terrified.

This time around my only significant concerns are blatant electoral fuckery, with Republicans trying to steal the election with voter suppression and court battles to invalidate votes.

Take those concerns away, and I'd be personally be 99% confident of a Biden victory.

[beachbumbob]

11. the 2016 polls were more correct than not and definitely NOT Wrong

in any way. The Comey hit was all it took to stop any Hillary momentum and in fact caused a reversal. In 2020 every element of this election is different from 2020

- Biden does not have high unfavorables
- is favorables are great
- he is NOT a flawed candidate
- there is NO fracture in the democratic base
- there is NO bernie bro movement, russian influence in social media has been a failure
- we have 4 years of trump's failures
- we democrats are on the hunt and 2018 is the outcome as will 2020
- we have trump lying and deflecting with every sentence he speaks and only his rabid base listens


this no 2016 and the outcome will reflect it

[DrToast]

8. The term "statistical tie" needs to die a horrible death and never be heard from again

There is no such thing.

https://washingtonmonthly.com/2008/08/13/margin-of-error/

[Fiendish Thingy]

10. You discounted the most important part:

Margins of error are typically stated for a 95% confidence interval. So this example 52±3% will be in the range 49-55 95% of the time. 5% of the time the real result could even be higher or lower than that ±3%. Only 2.5% of the time would Biden be at 49% or lower, Only 2.5% of the time would Biden be at 55% or higher. Typical results will cluster more towards the middle of the range.

Say that Trump is polling at 48% in the same poll. As O'Donnell explained it, he correctly pointed out that you have to apply the margin of error to both numbers, so even though 48% and 52% are 4% apart, the results overlap, with Trump possibly going as high as 51%, and Biden possibly going as low as 49%.

The scenario above is indeed an example of a “statistical tie”, and shouldn’t be discounted, especially when compared to a poll that isn’t a statistical tie.

For example, a poll from an A rated pollster that gives Biden a 9pt. Margin, but has a 5% MOE is a statistical tie (double the MOE to get the range of possible results between two candidates w/95% confidence). Contrast that with an A rated poll that has Biden with an 8pt. Margin, but only a 3% MOE- the latter represents a stronger, more stable representation of the actual outcome.

You said O’Donnell was wrong to point out the statistical tie - he was not, it was good journalism IMO.
Of course, following the trend/average will give the most accurate results overall, but one should not overlook the significance of the MOE.

[Silent3]

12. As some else pointed out, the term "statistical tie" is, in and of itself, a problem.

Whoever leads in a poll (given the poll is based on a good model) has a higher chance of winning. Period. No matter how narrow the lead, nor how wide the MoE, the odds never convert to 50/50 when you take the MoE into account.

If there were a well-understood meaning for "statistical tie", like "this poll only gives the leading candidate a 60% chance of winning", that would different. But there is no such well-understood meaning, so the terminology, especially in the hands of people who aren't good at communicating statistics, can be very misleading.

[Fiendish Thingy]

16. Would you agree that a lead outside the MOE is better than one inside the MOE?

Even if the lead outside the MOE is smaller than the one inside the MOE?

In Clinton’s 2016 Rust Belt polls, her lead was within MOE, and most, but not all, of Biden’s Rust Belt leads are outside the MOE (and thankfully much larger than Clinton’s, and most significantly, above 50%). Most of Biden’s other swing state/Sun Belt polls are within the MOE, so I expect those races will be closer than MI, WI and MN.

[Silent3]

18. Certainly it's better to be outside the MoE

And it's even better still to be five times outside the MoE.

That doesn't make the phrase "statistical tie", for something inside the MoE, any less misleading, however.

[Fiendish Thingy]

19. Nevertheless, there is a significant difference between leads outside/inside the MOE Nt

[Silent3]

20. Significant in what way?

The wider the spread between two candidates, the more certain the apparent leader will win. But there's no special, magical change that happens when you transition from inside the MoE to outside the MoE. It's a smooth continuum.

I don't know how to do the exact calculation off the top of my head, but if the MoE is ±3@95% for both polling values for two candidates, and their individual scores are 3% apart, the apparent leading candidate has something like an 80-90% chance of winning, because of the way the two bell curves interact.

[grantcart]

13. O'Donnell and all the others who conflate MOE and statistical tie are wrong, the OP is correct

Further O'Donnell, who I love, is not a journalist and doesn't exercise journalism. He is an opinion commentator, it is as different as reading the opinion page and the rest of a newspaper.

Using a poll of poll like 538 which also weighs pollster performance a candidate with a 52 - 47 lead and a 5% MOE isn't in a statistical tie.

There is as much chance for the outcome to be 56-44 for the leading candidate as it is for there to be a 50-50 tie.

In addition, if there have been 10 polls showing the lead candidate in the lead then there is significantly less chance that the MOE will move 100% to the lesser candidate.

[grantcart]

14. Excellent post

Morning Joe is another frequent "statistical tie" offender.


I would be very interested if you were interested in expanding the OP for a second edition on how a 538 poll of polls also narrows the MOE and how a race where one candidate has led in 10 straight polls would be even less likely to result in a tie when a 5% margin is within a 5% MOE.

[SharonClark]

15. Thank you! I'm bookmarking this post for future reference.

[treestar]

17. all pundits need it to be close

a dead heat. They want you to watch on election night.