Opinion Polls: Caveat Emptor

statsman

Well-known member
Joined
Feb 25, 2011
Messages
55,055
Every time there’s a new poll published, we on P.ie go into an entirely predictable pattern of posting:

Great, accurate poll. My party is doing well.
Crap, unreliable poll. My party is not doing well.

To a certain extent, this is understandable, but could be avoided by standing back and asking a few simple questions.

  • What questions are asked?
  • When?
  • What is the sample size?
  • How was the sample selected (random, quota, self-selecting)?
  • What was the mode (face-to-face, phone, e-mail, other)?
  • What is the margin of error?
  • Who did it?
  • Who paid for it?
If you want to know if a pot of soup needs more salt, you just need to stir it well and then taste a spoonful, not eat the whole pot. If you want to gauge the opinion of an entire population, you don’t have to ask everyone, just a representative sample. Quota samples will reflect the composition of the larger pot by pre-selecting quotas of groups to ask (gender, age, social class, geography, etc.) to fit the census breakdown along those lines. So, if 30% of the population are working-class women over 40, then the sample should reflect this. Random samples will just ask 1,000 (or whatever the sample size is going to be) people, and then adjust to reflect. So, if they find that only 200 of the sample are working-class women over 40 but it should be 220, then they weigh the responses of that 200 so that each of them counts as 1.1 person. Small representative samples will always be better than big, self-selecting ones.

Then they work out the margin of error (MoE), which is always an approximation as it is based on an ideal situation. For example, statistical theory confirms that for a sample of 1,000 with a 100% response rate on a question where the population is split 50/50, the poll will have an MoE of +/- 3% 19 times out of 20. Clearly, this never happens, but the theory acts as a guideline for an MoE that isn’t too far off.

Despite what some people think, genuine, serious polling companies have no interest in being wrong, so polls that are conducted by one of them are more likely to be right than polls conducted by randomwebsire.ru. Equally, most serious media organisations want to be as accurate as possible, so, like them or not, a poll commissioned by RTE is more likely to be accurate than one commissioned by a company that wants to sell a product.

So, why do polls on, say, voter intention in Ireland, that are published within a week of each other often show very different trends? We’re back to these questions:

  • What questions are asked?
  • When?
Clearly, the question asked will have a strong influence. Just think about the difference between being asked ‘Which party is currently doing the best job?’ and ‘Which party would you vote for if there were a GE tomorrow?

Equally, time plays a role. Would you expect the same response rate the day before a big scandal as you’d get the day after?

So, most polls are accurate enough to within the MoE for the questions that were asked at the time they were asked. If you don’t like the result, then the problem is probably more with you than with the poll. And always remember, polls are not predictors of election results, they are samples of how salty the soup is at the moment you test it.

If you’d like more detail, this link is very useful:

A Journalist
 
Last edited:


gerhard dengler

Well-known member
Joined
Feb 3, 2011
Messages
46,739
Good effort for an OP. It makes a pleasant change here to read an OP which shows some thought and effort.
 

hollandia

Well-known member
Joined
Sep 11, 2012
Messages
30,149
Every time there’s a new poll published, we on P.ie go into an entirely predictable pattern of posting:

Great, accurate poll. My party is doing well.
Crap, unreliable poll. My party is not doing well.

To a certain extent, this is understandable, but could be avoided by standing back and asking a few simple questions.

  • What questions are asked?
  • When?
  • What is the sample size?
  • How was the sample selected (random, quota, self-selecting)?
  • What was the mode (face-to-face, phone, e-mail, other)?
  • What is the margin of error?
  • Who did it?
  • Who paid for it?
If you want to know if a pot of soup needs more salt, you just need to stir it well and then taste a spoonful, not eat the whole pot. If you want to gauge the opinion of an entire population, you don’t have to ask everyone, just a representative sample. Quota samples will reflect the composition of the larger pot by pre-selecting quotas of groups to ask (gender, age, social class, geography, etc.) to fit the census breakdown along those lines. So, if 30% of the population are working-class women over 40, then the sample should reflect this. Random samples will just ask 1,000 (or whatever the sample size is going to be) people, and then adjust to reflect. So, if they find that only 200 of the sample are working-class women over 40 but it should be 220, then they weigh the responses of that 200 so that each of them counts as 1.1 person. Small representative samples will always be better than big, self-selecting ones.

Then they work out the margin of error (MoE), which is always an approximation as it is based on an ideal situation. For example, statistical theory confirms that for a sample of 1,000 with a 100% response rate on a question where the population is split 50/50, the poll will have an MoE of +/- 3% 19 times out of 20. Clearly, this never happens, but the theory acts as a guideline for an MoE that isn’t too far off.

Despite what some people think, genuine, serious polling companies have no interest in being wrong, so polls that are conducted by one of them is more likely to be right than polls conducted by randomwebsire.ru. Equally, most serious media organisations want to be as accurate as possible, so, like them or not, a poll commissioned by RTE is more likely to be accurate than one commissioned by a company that wants to sell a product.

So, why do polls on, say, voter intention in Ireland, that are published within a week of each other often show very different trends? We’re back to these questions:

  • What questions are asked?
  • When?
Clearly, the question asked will have a strong influence. Just think about the difference between being asked ‘Which party is currently doing the best job?’ and ‘Which party would you vote for if there were a GE tomorrow?

Equally, time plays a role. Would you expect the same response rate the day before a big scandal as you’d get the day after?

So, most polls are accurate enough to within the MoE for the questions that were asked at the time they were asked. If you don’t like the result, then the problem is probably more with you than with the poll. And always remember, polls are not predictors of election results, they are samples of how salty the soup is at the moment you test it.

If you’d like more detail, this link is very useful:

A Journalist
Excellent post. Unfortunately, on poll threads there is very little analysis of the actual numbers, nor the methodology, nor the consistent variances between the various polling companies some give a detailed party by party breakdown, others lump everyone else into "others" for example.

Because no one (or very few, I do and a few others do) actually analyses the trends, the swings etc, we get threads that descend into spamming, shilling and abuse.

Unfortunately, only one like is available for the OP. And as a reward for your effort, here's a nice graph:



(Note to self - add some trend lines by party for clarity next time)
 

PBP voter

Well-known member
Joined
Sep 18, 2015
Messages
9,254
People just need to remember that +or-3% can make a huge difference.

FF 25%
FG 25%
SF 14%

was the result last time.

Technically a polling company could produce this before the election and be with the margin of error.

FG 28%
FF 22%
SF 17%.

Yet the reactionary media and plebs who cannot or will not listen will be telling us how wrong they got it.
 

bokuden

Well-known member
Joined
Sep 23, 2010
Messages
11,250
I think a broad average taken across all major polls over a period of time can give a general idea of how parties are doing. Individual polls are pretty meaningless TBH.
 

Dame_Enda

Well-known member
Joined
Dec 14, 2011
Messages
53,666
Irish polls tend to be more accurate in elections but not at all accurate in referendums. And I say that as a poll-junkie.

I think the lesson of the US and UK elections are that White voters are voting along educational and to a lesser extent class lines. Meanwhile most non-White voters continue to primarily vote for the Left especially in the US because of the Lefts support for multiculturalism and pandering to Identity Politics. The state of US inner cities calls into question whether they are voting in their best interests though.
 
Last edited:

Clanrickard

Well-known member
Joined
Apr 25, 2008
Messages
33,160
Good effort for an OP. It makes a pleasant change here to read an OP which shows some thought and effort.
Makes a change from his meaningless bland one liners.
 

bokuden

Well-known member
Joined
Sep 23, 2010
Messages
11,250
People just need to remember that +or-3% can make a huge difference.

FF 25%
FG 25%
SF 14%

was the result last time.

Technically a polling company could produce this before the election and be with the margin of error.

FG 28%
FF 22%
SF 17%.

Yet the reactionary media and plebs who cannot or will not listen will be telling us how wrong they got it.
I think that's broadly accurate.
 

Levellers

Well-known member
Joined
Apr 30, 2011
Messages
14,071
Polls are not worth that much. The trend of polls over a period of say two years would be of some interest.

Polls can be manipulated as well as they were before the last British general election when they deliberately under estimated the Tory vote in order to get the Tory vote out.
 

Dame_Enda

Well-known member
Joined
Dec 14, 2011
Messages
53,666
Possibly due to low turnout in referendums.
The turnout in the Citizenship referendum was 60% and yet 80% voted yes compared to 52% in the last poll by Red C.
 

Round tower

Well-known member
Joined
Feb 9, 2011
Messages
7,294
People just need to remember that +or-3% can make a huge difference.

FF 25%
FG 25%

SF 14%

was the result last time.

Technically a polling company could produce this before the election and be with the margin of error.

FG 28%
FF 22%
SF 17%.

Yet the reactionary media and plebs who cannot or will not listen will be telling us how wrong they got it.
Are u saying that was the result of the last GE, if it is its not completley right as FG was a ahead of FG in % of the vote
 

blokesbloke

Well-known member
Joined
Jan 13, 2011
Messages
22,697
Makes a change from his meaningless bland one liners.
... said Clanrickard, in a meaningless bland one-liner post which had nothing do with the OP and was the first off-topic post of the thread... :roll:
 

former wesleyan

Well-known member
Joined
Nov 29, 2009
Messages
25,565

statsman

Well-known member
Joined
Feb 25, 2011
Messages
55,055
Excellent post. Unfortunately, on poll threads there is very little analysis of the actual numbers, nor the methodology, nor the consistent variances between the various polling companies some give a detailed party by party breakdown, others lump everyone else into "others" for example.

Because no one (or very few, I do and a few others do) actually analyses the trends, the swings etc, we get threads that descend into spamming, shilling and abuse.

Unfortunately, only one like is available for the OP. And as a reward for your effort, here's a nice graph:



(Note to self - add some trend lines by party for clarity next time)
I do love a nice graph.
 

Clanrickard

Well-known member
Joined
Apr 25, 2008
Messages
33,160

Clanrickard

Well-known member
Joined
Apr 25, 2008
Messages
33,160
... said Clanrickard, in a meaningless bland one-liner post which had nothing do with the OP and was the first off-topic post of the thread... :roll:
Well seen as statsman spends his time derailing threads he doesn't like I thought it appropriate.
 

gatsbygirl20

Well-known member
Joined
Dec 1, 2008
Messages
22,551
The turnout in the Citizenship referendum was 60% and yet 80% voted yes compared to 52% in the last poll by Red C.
There are perhaps too many polls of dubious quality coming thick and fast before elections, for example.

They have become part of the theatre--the circus--of elections, and no doubt help sell newspapers, etc.

Voters get caught up in the circus. I know I do, and my mathematical skills are not sufficient to break down the numbers or to really question what I am being fed. I am probably not unique in this.
 


New Threads

Popular Threads

Most Replies

Top