The Four Horsemen
of Validity
Session 6
PMAP 8521: Program evaluation
Andrew Young School of Policy Studies
Plan for today
Plan for today
Construct validity
Plan for today
Construct validity
Statistical conclusion validity
Plan for today
Construct validity
Statistical conclusion validity
Internal validity
Plan for today
Construct validity
Statistical conclusion validity
Internal validity
External validity
Construct validity
A new program hopes to
improve student commitment to school
A new program hopes to
improve student commitment to school
Participants score 200 points higher on the SAT and have a 0.3 higher GPA, on average
A new program hopes to
improve student commitment to school
Participants score 200 points higher on the SAT and have a 0.3 higher GPA, on average
Success! Success?
The Streetlight Effect
Drunk guy looking for keys in the light of the lamppost instead of over in the bushes where they lost them
Construct validity
Are you measuring what you want to measure?
Construct validity
Are you measuring what you want to measure?
Do test scores measure commitment to school?
Teacher performance? Principal skill?
Construct validity
Are you measuring what you want to measure?
Do test scores measure commitment to school?
Teacher performance? Principal skill?
Test scores measure how good kids are at taking tests
Construct validity
Are you measuring what you want to measure?
Do test scores measure commitment to school?
Teacher performance? Principal skill?
Test scores measure how good kids are at taking tests
This is why we spend so much time
on outcome measurement construction!
Statistical conclusion
validity
Statistical conclusion validity
Statistical conclusion validity
Are your statistics correct?
Statistical conclusion validity
Are your statistics correct?
Statistical power
Statistical conclusion validity
Are your statistics correct?
Statistical power
Violated assumptions of statistical tests
Statistical conclusion validity
Are your statistics correct?
Statistical power
Violated assumptions of statistical tests
Fishing and p-hacking
Statistical conclusion validity
Are your statistics correct?
Statistical power
Violated assumptions of statistical tests
Fishing and p-hacking
Spurious statistical significance
Power
A training program causes incomes to rise by $40
| Person | Group | Before | After | Difference |
|---|---|---|---|---|
| 295 | Control | 122.09 | 229.04 | 106.95 |
| 126 | Treatment | 205.60 | 199.84 | -5.76 |
| 400 | Control | 133.25 | 130.40 | -2.85 |
| 94 | Treatment | 270.11 | 206.56 | -63.54 |
| 250 | Control | 344.37 | 222.89 | -121.49 |
| 59 | Treatment | 312.41 | 268.06 | -44.35 |
Power
Survey 10 participants
Power
Survey 10 participants
Survey 200 participants
What's the right sample size?
Use a statistical power calculator to
make sure you can potentially detect an effect
Test assumptions
Every statistical test has certain assumptions
Test assumptions
Every statistical test has certain assumptions
For instance, for OLS:
Linearity Homoscedasticity Independence Normality
Test assumptions
Every statistical test has certain assumptions
For instance, for OLS:
Linearity Homoscedasticity Independence Normality
Make sure you're doing the stats correctly
Fishing and p-hacking
Wouldn't it be awesome to run thousands of models
with different combinations of variables
until you find coefficients that are statistically significant?
Fishing and p-hacking
Wouldn't it be awesome to run thousands of models
with different combinations of variables
until you find coefficients that are statistically significant?
Don't!
Fishing and p-hacking
Wouldn't it be awesome to run thousands of models
with different combinations of variables
until you find coefficients that are statistically significant?
Don't!
Spurious statistical significance
If p threshold is 0.05 and you measure 20 outcomes,
1 will likely show correlation by chance
Spurious statistical significance
If p threshold is 0.05 and you measure 20 outcomes,
1 will likely show correlation by chance
Internal validity
Internal validity
Internal validity
Omitted variable bias
Selection Attrition
Internal validity
Omitted variable bias
Selection Attrition
Trends
Maturation Secular trends Seasonality Testing Regression
Internal validity
Omitted variable bias
Selection Attrition
Trends
Maturation Secular trends Seasonality Testing Regression
Study calibration
Measurement error
Time frame
Internal validity
Omitted variable bias
Selection Attrition
Trends
Maturation Secular trends Seasonality Testing Regression
Study calibration
Measurement error
Time frame
Contamination
Hawthorne John Henry
Spillovers Intervening events
Selection
If people can choose to enroll in a
program, those who enroll will be
different from those who do not
Selection
If people can choose to enroll in a
program, those who enroll will be
different from those who do not
How to fix
Randomization into
treatment and control groups
Selection
If people can choose when to
enroll in a program, time might
influence the result
Selection
If people can choose when to
enroll in a program, time might
influence the result
How to fix
Shift time around
(happier people more likely to get married, so without randomly assigning marriage how would you study the impact of marriage on happiness?). They use a simple approach - since happiness varies over time set marriage equal to time zero and build a pre-post design around it. You essentially leverage the within-group variance and iron out across-age differences because of the varying ages of marriage. The whole insight it to change the time-line from calendar years to program years.
Attrition
If the people who leave a program or
study are different than those who stay,
the effects will be biased
Attrition
If the people who leave a program or
study are different than those who stay,
the effects will be biased
How to fix
Check characteristics of those
who stay and those who leave
Fake microfinance program results
| ID | Increase in income | Remained in program |
|---|---|---|
| 1 | $3.00 | Yes |
| 2 | $3.50 | Yes |
| 3 | $2.00 | Yes |
| 4 | $1.50 | No |
| 5 | $1.00 | No |
ATE with
attriters = $2.20
Fake microfinance program results
| ID | Increase in income | Remained in program |
|---|---|---|
| 1 | $3.00 | Yes |
| 2 | $3.50 | Yes |
| 3 | $2.00 | Yes |
| 4 | $1.50 | No |
| 5 | $1.00 | No |
ATE with
attriters = $2.20
ATE without
attriters = $2.83
Maturation
Growth is expected naturally
e.g. programs targeted at childhood development
contend with the fact that children develop on their own too
Maturation
Growth is expected naturally
e.g. programs targeted at childhood development
contend with the fact that children develop on their own too
How to fix
Use a comparison group to remove the trend
Secular trends
Patterns in data happen
because of larger global processes
Recessions Cultural shifts Marriage equality
Secular trends
Patterns in data happen
because of larger global processes
Recessions Cultural shifts Marriage equality
How to fix
Use a comparison group to remove the trend
Seasonal trends
Patterns in data happen because of
regular time-based trends
Seasonal trends
Patterns in data happen because of
regular time-based trends
How to fix
Compare observations from same time period
or use yearly/monthly averages
Testing
Repeated exposure to questions or tasks
will make people improve naturally
Testing
Repeated exposure to questions or tasks
will make people improve naturally
How to fix
Change tests, maybe don't offer pre-tests,
use a control group that receives the test
Regression to the mean
People in the extreme have a tendency to
become less extreme over time
Luck Crime and terrorism Hot hand effect
Regression to the mean
People in the extreme have a tendency to
become less extreme over time
Luck Crime and terrorism Hot hand effect
How to fix
Don't select super high or
super low performers
This isn’t because the universe trends toward some average; an extreme value is because of systematic and random extremes, which are rare. Luck goes away
Measurement error
Measuring the outcome incorrectly
will bias the effect
Measurement error
Measuring the outcome incorrectly
will bias the effect
How to fix
Measure the outcome well
Time frame
If the study is too short, the effect might not
be detectable yet; if the study is too long,
attrition becomes a problem
Time frame
If the study is too short, the effect might not
be detectable yet; if the study is too long,
attrition becomes a problem
How to fix
Use prior knowledge about the thing
you're studying to choose the right length
Hawthorne effect
Observing people makes them behave differently
Hawthorne effect
Observing people makes them behave differently
How to fix
Hide? Use completely unobserved control groups
Experiments in 1924-1932 at Hawthorne Works
John Henry effect
Control group works hard to prove
they're as good as the treatment group
John Henry effect
Control group works hard to prove
they're as good as the treatment group
How to fix
Keep two groups separate
Spillover effect
Control groups naturally pick up
what the treatment group is getting
Externalities Social interaction Equilibrium effects
Spillover effect
Control groups naturally pick up
what the treatment group is getting
Externalities Social interaction Equilibrium effects
How to fix
Keep two groups separate;
use distant control groups
Intervening events
Something happens that affects one of
the groups and not the other
Intervening events
Something happens that affects one of
the groups and not the other
How to fix
🤷♂️
Internal validity
Omitted variable bias
Selection Attrition
Trends
Maturation Secular trends Seasonality Testing Regression
Study calibration
Measurement error
Time frame
Contamination
Hawthorne John Henry
Spillovers Intervening events
Fixing internal validity
Randomization fixes a host of issues
Selection Maturation Regression to the mean
Fixing internal validity
Randomization fixes a host of issues
Selection Maturation Regression to the mean
Randomization doesn't fix everything!
Attrition Contamination Measurement
External validity
Generalizability
Are your findings generalizable
to the whole population?
Generalizability
Are your findings generalizable
to the whole population?
Generalizability
Are your findings generalizable
to the whole population?
Lab conditions vs. real world
Study volunteers are weird
Lab conditions vs. real world
Study volunteers are weird
Western, educated, from industrialized,
rich, and democratic countries
Lab conditions vs. real world
Study volunteers are weird
Western, educated, from industrialized,
rich, and democratic countries
Not everyone takes surveys
Lab conditions vs. real world
Study volunteers are weird
Western, educated, from industrialized,
rich, and democratic countries
Not everyone takes surveys
Online surveys Amazon Mechanical Turk Random digit dialing
Different settings and circumstances
Different settings and circumstances
Does a study in one state
apply to other states?
Different settings and circumstances
Does a study in one state
apply to other states?
Does the effect from a mosquito net trial
in Eritrea transfer to Bolivia?