Why polls? Why forecasting?

1. Journalism

1a. Attention

1b. Truth

2. Methods training

3. It's fun!

So let's talk about...

1. How polls work

2. How forecassts work

3. And why they fail

The first polls

Straw polls -> real polls

The first ("scientific") polls

- Conducted face-to-face

- Used demographic quotas for representativeness

• Race, gender, age, geography, class

- Beat straw polls in accuracy (1936)

• By shrinking bias from nonresponse

- But fell short of true survey science (1948)

Polls 2.0

- SSRC says: area sampling

Polls 2.0

- SSRC says: area sampling

- Gallup implements some partisan controls

• Stratas are groups of precincts by 1948 vote choice

- Use rough quotas within geography

- Preserve interviewer bias

Polls 3.0

Technological change -> better methods

Polls 3.0

- True random sampling (for people with phones)

- Response rates above 70 or 80%

- Rarer instances of severe nonresponse bias

- Cheaper to conduct = news orgs poll (CBS, NYT)

Technological change -> worse methods?

The soup principle (in theory)

Image credit: Pew Research Center

- People could be very dissimilar by group, meaning small deviations in sample demographics cause big errors (sampling error)

- Or the people who respond to the poll could be systematically different from the people who don't (response error)

- Or your list of potential respondents could be missing people (coverage error)

*Polls can also go wrong if they have bad question wording, a fourth type of survey error called "measurement error"

The soup principle (in practice)

So, to satisfy the soup principle...

Pollsters use statistical algorithms to ensure their samples match the population on different demographic targets

• Race, age, gender, and region are most common

Option A: Weighting

• Raking, calibration, propensity etc.

Option B: Modeling

2016: Education weighting

2020: Partisan nonresponse

The future of polling?

1. More weighting variables

2. More online and off-phone data colleciton (SMS, mail)

3. Mixed samples

All in the pursuit of getting representative (and politically balanced) samples before the adjustment stage

2020 presidential election forecast

What goes into the model?

1. National economic + political fundamentals

2. Decompose into state-level priors

3. Polls

Uncertainty is propogated throughout the models, incorporated via MCMC sampling in step 3.

National fundamentals?

i) Index of economic growth (1940 - 2016)

• eight different variables, scaled to measure the standard-deviation from average annual growth

ii) Presidential approval (1948 - 2016)

iii) Polarization (1948 - 2016)

• measured as the share of swing voters in the electorate, per the ANES --- and interacted with economic growth

iv) Whether an incumbent is on the ballot

Modeling the fundamentals

Model formula:

vote ~ incumbent_running:economic growth:polarization + approval

Training

Model trained on 1948-2016 using elastic net regression with leave-one-out cross-validation

RMSE = 2.6 percentage points on two-party Democratic vote share

The model is a federalist

i) Train a model to predict the Democratic share of the vote in a state relative to the national vote, 1948-2016

• Variables are: lean in the last election, lean two elections ago, home state effects * state size, conditional on the national vote in the state

ii) Use the covariates to make predictions for 2020, conditional on the national fundamentals prediction for every day

ii) Simulate state-level outcomes to extract a mean and standard deviation

• Propogates uncertainty both from the LOOCV RMSE of the national model and the state-level model

Just a trend through points...

Can do with any series of packages for R, other statistical languages

(...but with some fancy extra stuff)

mu_b[:,T] = cholesky_ss_cov_mu_b_T * raw_mu_b_T + mu_b_prior;  
for (i in 1:(T-1)) mu_b[:, T - i] = cholesky_ss_cov_mu_b_walk * raw_mu_b[:, T - i] + mu_b[:, T + 1 - i];
national_mu_b_average = transpose(mu_b) * state_weights;
mu_c = raw_mu_c * sigma_c;
mu_m = raw_mu_m * sigma_m;
mu_pop = raw_mu_pop * sigma_pop;
e_bias[1] = raw_e_bias[1] * sigma_e_bias;
sigma_rho = sqrt(1-square(rho_e_bias)) * sigma_e_bias;
for (t in 2:T) e_bias[t] = mu_e_bias + rho_e_bias * (e_bias[t - 1] - mu_e_bias) + raw_e_bias[t] * sigma_rho;
//*** fill pi_democrat
for (i in 1:N_state_polls){
  logit_pi_democrat_state[i] = 
    mu_b[state[i], day_state[i]] + 
    mu_c[poll_state[i]] + 
    mu_m[poll_mode_state[i]] + 
    mu_pop[poll_pop_state[i]] + 
    unadjusted_state[i] * e_bias[day_state[i]] +
    raw_measure_noise_state[i] * sigma_measure_noise_state + 
    polling_bias[state[i]];
}

Poll-level model

i. Latent state-level vote shares evolve as a random walk over time

• Pooling toward the state-level fundamentals more as we are further out from election day

ii. Polls are observations with measurement error that are debiased on the basis of:

• Pollster firm (so-called "house effects")
• Poll mode
• Poll population

iii. Correcting for partisan non-response

• Whether a pollster weights by party registration or past vote
• (Incorporated as a residual AR process)

Debiased predictions

Notable improvements from partisan non-responseand other weighting issues

Debiased predictions

Notable improvements* from controlling for partisan nonresponse and other weighting issues

*In 2016, but not 2020

If polls are biased, the aggregation model cannot remove the biases. It can only explore them

Yet this means violation of random sampling also violates the statistical theory underpinning election models!

And means we must add ways to incorporate the extra uncertainty from bias. (Especially if partisan nonresponse is getting worse.)

Some ideas...

1. Add extra model error for uncertainty. But how?

2. Attempt to debias polls (but that adds uncertainty too)

3. Only use polls we trust? (But how do we measure trust? Not necessarily a robust solution.)

4. Communication: Show forecasts for different scenarios of poll error

Ultimately, we're still figuring out the answer....

These slides were made using the xaringan package for R. They are available online at https://www.gelliottmorris.com/slides/