+ - 0:00:00
Notes for current slide
Notes for next slide

What’s the matter with polling?

From Strength in Numbers: How Polls Work + Why We Need Them

G. Elliott Morris | October 13, 2022 | Berkeley, CA

1 / 38

2 / 38

3 / 38

The "soup principle"

4 / 38

The first polls

5 / 38

"Straw" polls

6 / 38

7 / 38

8 / 38

The first ("scientific") polls

- Conducted face-to-face

9 / 38

The first ("scientific") polls

- Conducted face-to-face

- Used demographic quotas for representativeness

  • Race, gender, age, geography
9 / 38

The first ("scientific") polls

- Conducted face-to-face

- Used demographic quotas for representativeness

  • Race, gender, age, geography

- Beat straw polls in accuracy (1936)

  • By shrinking bias from demographic nonresponse
9 / 38

The first ("scientific") polls

- Conducted face-to-face

- Used demographic quotas for representativeness

  • Race, gender, age, geography

- Beat straw polls in accuracy (1936)

  • By shrinking bias from demographic nonresponse
9 / 38

The first ("scientific") polls

- But fell short of true survey science (1948)

10 / 38

Polls 2.0

- SSRC says: area sampling

11 / 38

Polls 2.0

- SSRC says: area sampling

11 / 38

Polls 2.0

- SSRC says: area sampling

- Gallup implements some partisan controls

  • Strata are groups of precincts by 1948 vote choice
12 / 38

Polls 2.0

- SSRC says: area sampling

- Gallup implements some partisan controls

  • Strata are groups of precincts by 1948 vote choice

- Use rough quotas within geography

12 / 38

Polls 2.0

- SSRC says: area sampling

- Gallup implements some partisan controls

  • Strata are groups of precincts by 1948 vote choice

- Use rough quotas within geography

- But, preserve interviewer bias

12 / 38

Polls 2.0

- SSRC says: area sampling

- Gallup implements some partisan controls

  • Strata are groups of precincts by 1948 vote choice

- Use rough quotas within geography

- But, preserve interviewer bias

12 / 38

Polls 3.0

13 / 38

Polls 3.0

Technological change -> better methods

13 / 38

Polls 3.0

- 1970s: true random sampling (for people with phones)

- Response rates above 70-80%

- Rarer instances of severe nonresponse bias

- Cheaper to conduct = many news orgs poll (CBS, NYT)

14 / 38

15 / 38

The soup principle: satisfied?

Source: Pew Research Center

16 / 38

The soup principle: satisfied?

1. RDD polls are representative (at high response)

2. Availability of many different surveys allow for extra layer of aggregation to control for choices made by individual researcheers

17 / 38

= perfect polls forever,

18 / 38

= perfect polls forever,

...right?

18 / 38

Technological change -> worse methods?

Source: Pew Research Center

19 / 38

Polarized voting -> harder sampling

Source: Webster & Abramowitz 2017

20 / 38

But what if the people you sample don't represent the population?

21 / 38

But what if the people you sample don't represent the population?

- People could be very dissimilar by group, meaning small deviations in sample demographics cause big errors (sampling error)

21 / 38

But what if the people you sample don't represent the population?

- People could be very dissimilar by group, meaning small deviations in sample demographics cause big errors (sampling error)

- Or the people who respond to the poll could be systematically different from the people who don't (response error)

21 / 38

But what if the people you sample don't represent the population?

- People could be very dissimilar by group, meaning small deviations in sample demographics cause big errors (sampling error)

- Or the people who respond to the poll could be systematically different from the people who don't (response error)

- Or your list of potential respondents could be missing people (coverage error)

21 / 38

But what if the people you sample don't represent the population?

- People could be very dissimilar by group, meaning small deviations in sample demographics cause big errors (sampling error)

- Or the people who respond to the poll could be systematically different from the people who don't (response error)

- Or your list of potential respondents could be missing people (coverage error)

 

 

*Polls can also go wrong if they have bad question wording, a fourth type of survey error called "measurement error"

21 / 38

The soup principle in theory

Source: Pew Research Center

22 / 38

The soup principle in practice

23 / 38

Polls today are not soup

 

24 / 38

Polls today are not soup

 

- Declining response rates + Internet = innovations in polling online, but they don't use random sampling

24 / 38

Polls today are not soup

 

- Declining response rates + Internet = innovations in polling online, but they don't use random sampling

- And even traditional RDD polls don't have a true random sample (since response rates are too low)

- And because of nonresponse

24 / 38

So, to satisfy the soup principle...

Pollsters use statistical algorithms to ensure their samples match the population on different demographic targets

  • Race, age, gender, and region are most common

  • Variety of methods (weighting, modeling) available

25 / 38

These adjustments make polls pretty good!

26 / 38

But in close races, they aren't enough:

27 / 38

2016: Education weighting

28 / 38

2020: Partisan nonresponse

29 / 38

2020: Partisan nonresponse

30 / 38

2020: Partisan nonresponse

  • Problem reaching Trump voters overall

30 / 38

2020: Partisan nonresponse

  • Problem reaching Trump voters overall

  • And within demographic groups

30 / 38

2020: Partisan nonresponse

  • Problem reaching Trump voters overall

  • And within demographic groups

  • Something you cannot fix with weighting

30 / 38

2020: Partisan nonresponse

  • Problem reaching Trump voters overall

  • And within demographic groups

  • Something you cannot fix with weighting

    • Pollsters can adjust for past vote, but the electorate changes, and certain types of eg Trump voters may not respond to surveys

30 / 38

Polls and soup in 2022

31 / 38

Polls and soup in 2022



A few ways forward:

31 / 38

Making polls work again

32 / 38

Making polls work again

1. More weighting variables (NYT)

32 / 38

Making polls work again

1. More weighting variables (NYT)

2. More online and off-phone data colleciton (SMS, mail)

32 / 38

Making polls work again

1. More weighting variables (NYT)

2. More online and off-phone data colleciton (SMS, mail)

3. Mixed samples (private pollsters)

32 / 38

Making polls work again

1. More weighting variables (NYT)

2. More online and off-phone data colleciton (SMS, mail)

3. Mixed samples (private pollsters)

In the pursuit of getting representative (and politically balanced) samples before and after the adjustment stage

32 / 38

In the pursuit of getting representative (and politically balanced) samples before and after the adjustment stage

33 / 38

In the pursuit of getting representative (and politically balanced) samples before and after the adjustment stage

To satisfy the soup principle

33 / 38

Further questions:

34 / 38

What if that doesn't work?

2022 a critical test: does surveys get better or stay the same — or do they get worse?

What if the DGP remains biased?

What if the quality of the average poll continues to fall?

35 / 38

Can we trust polls to be precise in close elections?

If not, what are they good for?

36 / 38

How Polls Work and Why We Need Them

37 / 38

Thank you!

STENGTH IN NUMBERS is Now available.



Website: gelliottmorris.com

Twitter: @gelliottmorris

Questions?

These slides were made using the xaringan package for R. They are available online at https://www.gelliottmorris.com/slides/

38 / 38

2 / 38
Paused

Help

Keyboard shortcuts

↑, ←, Pg Up, k Go to previous slide
↓, →, Pg Dn, Space, j Go to next slide
Home Go to first slide
End Go to last slide
Number + Return Go to specific slide
b / m / f Toggle blackout / mirrored / fullscreen mode
c Clone slideshow
p Toggle presenter mode
t Restart the presentation timer
?, h Toggle this help
Esc Back to slideshow