Polling aggregation in an era of high (potential) bias
From Strength in Numbers: How Polls Work + Why We Need Them
G. Elliott Morris | Oct 19 2022 | NYAAPOR
On average, error in polls is low
But a good aggregate needs polls with low bias,
But a good aggregate needs polls with low bias,
especially at the state level.
But a good aggregate needs polls with low bias,
especially at the state level.
Problem: bias in polls has been increasing
Problem: bias in polls has been increasing...
... and bias is correlated across levels and states
Solution 1: Less biased polls!
Less biased polls
Less biased polls
- Election-year partisan non-response bias is present within both demographic and lagged partisan groups (party ID, past vote, approval)
Less biased polls
Election-year partisan non-response bias is present within both demographic and lagged partisan groups (party ID, past vote, approval)
Something you cannot fix with standard weighting.
Less biased polls
Election-year partisan non-response bias is present within both demographic and lagged partisan groups (party ID, past vote, approval)
Something you cannot fix with standard weighting.
So...
Less biased polls
Election-year partisan non-response bias is present within both demographic and lagged partisan groups (party ID, past vote, approval)
Something you cannot fix with standard weighting.
So...
Options:
Less biased polls
Election-year partisan non-response bias is present within both demographic and lagged partisan groups (party ID, past vote, approval)
Something you cannot fix with standard weighting.
So...
Options:
- More weighting variables (NYT)
Less biased polls
Election-year partisan non-response bias is present within both demographic and lagged partisan groups (party ID, past vote, approval)
Something you cannot fix with standard weighting.
So...
Options:
More weighting variables (NYT)
More offline and off-phone data collection (Pew NPORS, SSRS, NORC)
Less biased polls
Election-year partisan non-response bias is present within both demographic and lagged partisan groups (party ID, past vote, approval)
Something you cannot fix with standard weighting.
So...
Options:
More weighting variables (NYT)
More offline and off-phone data collection (Pew NPORS, SSRS, NORC)
Mixed-mode samples (promising, but not yet popular among public pollsters)
Solution 1: Less biased polls!
Solution 1: Less biased polls!
or
Solution 1: Less biased polls!
or
Solution 1: Less biased polls?
"Solution" 2: Let the aggregation model debias the polls
Case study: Economist model
Case study: Economist model
i. Latent state-level vote shares evolve as a random walk over time
Case study: Economist model
i. Latent state-level vote shares evolve as a random walk over time
ii. Polls are weighted by their historical error and bias
- Based on past relationship between a pollster's lagged historical bias and performance of the aggregate
Case study: Economist model
i. Latent state-level vote shares evolve as a random walk over time
ii. Polls are weighted by their historical error and bias
- Based on past relationship between a pollster's lagged historical bias and performance of the aggregate
iii. Polls are observations with constant random effects to "debias" based on:
- Pollster firm (so-called "house effects")
- Poll mode
- Poll population
Case study: Economist model
i. Latent state-level vote shares evolve as a random walk over time
ii. Polls are weighted by their historical error and bias
- Based on past relationship between a pollster's lagged historical bias and performance of the aggregate
iii. Polls are observations with constant random effects to "debias" based on:
- Pollster firm (so-called "house effects")
- Poll mode
- Poll population
iv. Polls are also adjusted for potential partisan non-response
- Each poll has a covariate for whether it weights by party registration or past vote
- Effect is allowed to change over time
- Adjusts for biases that remain AFTER removing the other biases
Notable improvements!
In 2016...
In 2016...
... But not 2020
In 2016...
... But not 2020
The problem with solution 2:
The problem with solution 2:
1. Pollsters change their methods
The problem with solution 2:
1. Pollsters change their methods
2. Not all adjustments work
Solution 3: Conditional forecasting!
Solution 3: Conditional forecasting!
Solution 3: Conditional forecasting!
- Present aggregates assuming some amount of polling bias.
Solution 3: Conditional forecasting!
- Present aggregates assuming some amount of polling bias.
- As a way to explain to readers how bias enters the process of polling
Solution 3: Conditional forecasting!
- Present aggregates assuming some amount of polling bias.
- As a way to explain to readers how bias enters the process of polling
- And what happens to forecasts if bias now does not follow historical distributions
Conditional forecasting:
Conditional forecasting:
1. Debias polls
Conditional forecasting:
1. Debias polls
2. Rerun simulations
2. Rerun simulations
2. Rerun simulations
Advantage: leaves readers with a much clearer picture of possibilities for election outcomes if past patterns of bias aren't predictive of bias now (2016, 2020)
We will see if this helps...
Further questions:
What if that doesn't work?
2022 a critical test: does surveys get better or stay the same — or do they get worse?
What if the DGP remains biased?
What if the quality of the average poll continues to fall?
Can we trust polls to be precise in close elections?
If not, what are they good for?
How Polls Work and Why We Need Them
Thank you!
STENGTH IN NUMBERS is available now.
Website: gelliottmorris.com
Twitter: @gelliottmorris
Questions?
These slides were made using the xaringan package for R. They are available online at https://www.gelliottmorris.com/slides/