Opinion polls are extremely common and important in democracies, but not many people understand them. For example, look at this recent poll from ABC, showing that a declining number of Americans is happy about the situation in Iraq. For example, 54 % (down from 61 % since last month) thinks the war in Iraq was worth fighting.
On the bottom of the page, it says:
This ABCNEWS poll was conducted by telephone Sept. 4-7 among a random national sample of 1,004 adults. The results have a three-point error margin.
Now, you can ask, how on earth can anyone be certain that asking only one thousand people is sufficient to know within an error margin of only 3 percentage points how ~200 million Americans think?
It is important to realise that when a poll gives an error margin, it is obviously not absolutely certain that there is no more than 3 % difference between the opinion of the whole population and the people asked (the "sample"). It is at least theoretically possible to imagine that they just happened to ask the 1004 people with a specific opinion, and the vast majority disagreed with them.
So what do they mean? Enter level of confidence. Those who conduct a poll require a specific level of confidence. And they set the level of confidence, typically to 95% (or 90%). When the level of confidence is 95%, it means if you asked 1000 different people 1000 times (or whatever), 95% of these samples would fall within the margin of error, in this case 3 %. If you insist, you can calculate the probability that the real error was, say, 10 % (hint: it will not be high).
If you have a look at the actual formula and tables for confidence and margin of error, you will find that the intuitive reaction, that 1000 people can't possibly represent 200+ million, is wrong. When the whole population you want to sample reaches a certain size (and actually the size of a small city is sufficient) then you only have to think about what sample size gives which margin of error. Whether your population is a city with 100K people or the entire United States with ~200M adults, the error of margin is essentially the same. Actually, it is more important that the answers hover around 50%, since the numbers get more dodgy when there is only a small minority with specific opinions.
The real error margin comes into play if there is any bias in the sampling, but that is beyond this little explanation. We simply have to trust that the pollers in this case know their job and didn't just ask whoever was answering the phone at 2 PM on a given day.
So, asking 1004 people what they mean about Bush's policies in Iraq gives no more than 3 percentage points error margin at a 95% level of confidence.
Diclaimer: I am not a mathematician or statistician. I studied this in college years ago, and had forgotten most of it, but I still hope I got it right with a little refresher course from the "I feel lucky" course at Google university.
