Evaluation and the causal revolution

# Evaluation and the causal revolution

**Session 1**

]

---

# Plan for today

.box-5.medium.sp-after-half[Data science and public service]

.box-3.medium.sp-after-half[Evidence, evaluation, and causation]

.box-6.medium.sp-after-half[Class details]

---

layout: false
name: ds-ps
class: center middle section-title section-title-5 animated fadeIn

# Data science and public service

---

# Data and government

.pull-left-3.center[
<figure>
 <img src="img/01/usds.png" alt="US Digital Service" title="US Digital Service" width="90%">
</figure>
]

.pull-middle-3.center[
<figure>
 <img src="img/01/dj-patil.jpg" alt="DJ Patil" title="DJ Patil" width="100%">
 <figcaption>DJ Patil</figcaption>
</figure>
]

???

DJ Patil, ["What Makes a Radical and Revolutionary Technology?](https://www.youtube.com/watch?v=UOoMzaWOQJA)

---

.center[
<figure>
 <img src="img/01/ddji.png" alt="Obama's Data-Driven Justice Initiative" title="Obama's Data-Driven Justice Initiative" width="70%">
</figure>
]

???

11.4 million people + 23 days

Data shared between agencies - arresting officer checks database, sees if person has been cycling, sees if they need mental health services

Two jails in Florida closed after this

https://obamawhitehouse.archives.gov/the-press-office/2016/06/30/fact-sheet-launching-data-driven-justice-initiative-disrupting-cycle

---

.center[
<figure>
 <img src="img/01/nola.jpg" alt="New Orleans hackathon" title="New Orleans hackathon" width="70%">
</figure>
]

---

???

http://us-cities.survey.okfn.org/

---

???

http://us-cities.survey.okfn.org/place/atlanta

---

???

https://datasetsearch.research.google.com/

---

.box-5.large[How do you use all this data to make the world better?]

---

.box-5.large[Analyze it and uncover insights!]

---

---

# What is "statistics"?

&nbsp;

.box-inv-5.medium[Collecting and analyzing data from a representative sample in order to make inferences about a whole population]

---

# What is "data science"?

.box-inv-5.medium.sp-after[Turning raw data into understanding, insight, and knowledge]

---

# What's the difference?

.pull-left.center[
<figure>
 <img src="img/01/ds-venn.png" alt="Data science Venn diagram" title="Data science Venn diagram" width="90%">
</figure>
]

.box-inv-5.medium[Analyze]

.box-5.medium[Communicate]
]

---

# What is "program evaluation"?

.box-inv-5.medium[Measuring the effect of social programs on society]

---

layout: false
name: evidence-evaluation-causation
class: center middle section-title section-title-3 animated fadeIn

# Evidence, evaluation, and causation

---

---

# Evidence-based medicine

.center[
<figure>
 <img src="img/01/salk.jpg" alt="Salk trials" title="Salk trials" width="60%">
</figure>
]

???

Polio shots administered blindly with placebo - kids didn't know, doctor didn't know (double blind) - effect found

Why not just use those who didn't get a shot as the control group? Why placebo? Kids who opted out were poor (which actually had a lower incidence rate), so you can't compare them with the rich (which had higher incidence rate)

All drugs get the safety and efficacy requirement and double blind RCTs after that

https://www.npr.org/sections/health-shots/2020/05/22/860789014/the-race-for-a-polio-vaccine-differed-from-the-quest-to-prevent-coronavirus

---

# Modern evidence-based medicine

.box-inv-3.medium[Apply evidence to clinical treatment decisions]

.box-inv-3.medium.sp-after[Move away from clinical judgment and "craft knowledge"]

.box-3.medium[Is this good?]

???

No, this isn’t the greatest. Push for evidence-based practice can crowd out clinical judgment, which is valuable – we can’t ignore it or dismiss it

Doctors have seen things in the past. Lady Sybil in Downton Abbey and preeclampsia - family doctor knew more about the condition and Sybil—he had intuition and experience—but he was ignored

---

# Evidence-based policy

.box-inv-3.medium[RAND health insurance study]

.box-inv-3.medium[Oregon Medicaid expansion]

.box-inv-3.medium[HUD's Moving to Opportunity]

.box-inv-3.medium[Tennessee STAR]

???

- RAND: In a large-scale, multiyear experiment, participants who paid for a share of their health care used fewer health services than a comparison group given free care. It concluded that cost sharing reduced "inappropriate or unnecessary" medical care (overutilization), but also reduced "appropriate or needed" medical care.
- Oregon: ongoing
- MTO: Researchers found that voucher recipients lived in lower-crime neighborhoods and generally had better units than the control group families, but the experiment had no impact on earnings or educational attainment. It did, however, have unexpected results in health and happiness. Parents in families who moved to low-poverty areas had lower rates of obesity and depression, and positive impacts on behavior and outlook among young women (but not young men)
- STAR: smaller class sizes lead to better outcomes

---

# Policy evidence industry

.box-inv-3.medium[Jameel Poverty Action Lab (J-PAL)]

.box-inv-3.medium[Campbell Collaboration]

???

- https://www.povertyactionlab.org/
- https://campbellcollaboration.org/

---

&nbsp;

.box-3.large[Should we have evidence for every policy or program?]

.box-inv-3.medium[**No!**]

---

.pull-left.center[
<figure>
 <img src="img/01/ellie-smoking.png" alt="No RCTs for smoking" title="No RCTs for smoking" width="100%">
</figure>
]

.pull-right.center[
<figure>
 <img src="img/01/drink-id.jpg" alt="Drink ID sign" title="Drink ID sign" width="70%">
</figure>
]

???

https://twitter.com/epiellie/status/1017622949799571456?s=12

In the 1970s, experts were saying that reducing the drinking age would be fine - 26 states did - created a natural quasi-experiment for researchers - Cook then found that highway fatalities increase by 10% for 18–20-year-olds. That wouldn't have been possible without legislators inadvertently performing the experiment, paving the way for better science

So knowledge flows between the two sides - use innovation from administrators as the basis for better science

---

&nbsp;

.box-3.large[Where does program evaluation fit with all this?]

---

---

# Types of evaluation

.pull-left.center[
<figure>
 <img src="img/01/rossi.jpg" alt="Rossi, Evaluation" title="Rossi, Evaluation" width="75%">
</figure>
]

---

.center[
<figure>
 <img src="img/01/psd-logic-model.png" alt="PSD logic model" title="PSD logic model" width="70%">
</figure>
]

---

---

# Theories of change

.center[
<figure>
 <img src="img/01/psd-impact-theory.png" alt="PSD impact theory" title="PSD impact theory" width="100%">
</figure>
]

---

.center[
<figure>
 <img src="img/01/wb-eval.jpg" alt="Impact Evaluation in Practice" title="Impact Evaluation in Practice" width="40%">
</figure>
]

---

# Theory → impact

---

---

.center[
<figure>
 <img src="img/01/godwins-law.png" alt="Godwin's Law" title="Godwin's Law" width="50%">
</figure>
]

???

https://en.wikipedia.org/wiki/Godwin%27s_law

---

# Godwin's Law for statistics

.box-inv-3.medium[Correlation does not imply causation]

---

.center[
<figure>
 <img src="img/01/slate-title.png" alt="Slate headline" title="Slate headline" width="80%">
</figure>

<hr>

<figure>
 <img src="img/01/slate-quote.png" alt="Slate quote" title="Slate quote" width="80%">
</figure>
]

???

https://slate.com/technology/2012/10/correlation-does-not-imply-causation-how-the-internet-fell-in-love-with-a-stats-class-cliche.html

---

---

# Correlation vs. causation

---

# Correlation vs. causation

---

# Correlation vs. causation

---

.center[
<figure>
 <img src="img/01/john-tweets.png" alt="John Holbein on causality" title="John Holbein on causality" width="70%">
</figure>
]

---

---

# How does we know if **X** causes **Y**?

.box-inv-3.medium[**X** causes **Y** if…]

.box-inv-3.medium[…we intervene and change **X** without changing anything else…]

.box-inv-3.medium[…and **Y** changes]

---

# **Y** "listens to" **X**

.box-inv-3["A variable **X** is a cause of a variable **Y** if **Y** in any way relies on **X** for its value.… **X** is a cause of **Y** if **Y** listens to **X** and decides its value in response to what it hears" (Pearl, Glymour, and Jewell 2016, 5–6)]

.box-inv-3[A light switch causes a light to go on, but not if the bulb is burned out (no **Y** despite **X**), or if the light was already on (**Y** without **X**)]

---

# Causal relationships?

.box-inv-3.medium[Lighting fireworks causes noise]

.box-inv-3.medium[Rooster crows cause the sunrise]

.box-inv-3.medium[Getting an MPA/MPP increases your earnings]

.box-inv-3.medium[Colds go away a few days after you take vitamin C]

---

# Causation

.box-inv-3.medium[Causation = Correlation + time order + nonspuriousness]

---

# The causal revolution

.pull-left-narrow[
<figure>
 <img src="img/01/book-of-why.jpg" alt="The Book of Why" title="The Book of Why" width="100%">
</figure>
]

.pull-right-wide.center[
<figure>
 <img src="img/01/judea-pearl.jpg" alt="Judea Pearl" title="Judea Pearl" width="90%">
</figure>
]

---

# Causal diagrams

.box-inv-3.medium[Directed acyclic graphs (DAGs)]

.box-3.SMALL[Graphical model of the process that generates the data]

.box-3.SMALL[Maps your philosophical model]

.box-3.SMALL[Fancy math ("*do*-calculus") tells you what to control for to isolate and identify causation]

]

]

---

---

---

---

.center[
<figure>
 <img src="img/01/dagitty-model.png" alt="FASD DAG" title="FASD DAG" width="60%">
 <figcaption>Effect of smoking on fetal alcohol syndrome</figcaption>
</figure>
]

???

https://link.springer.com/article/10.1007/s00787-018-1264-3

---

???

https://twitter.com/andrewheiss/status/1208836534285340673

---

???

https://twitter.com/andrewheiss/status/1208836534285340673

---

layout: false
name: class-details
class: center middle section-title section-title-6 animated fadeIn

# Class details

---

---

# Plan for the class

.center[
<figure>
 <img src="img/01/eval-flowchart.png" alt="Class flowchart" title="Class flowchart" width="100%">
</figure>
]

---

# Textbooks

.pull-left-3.center[
<figure>
 <img src="img/01/wb-eval.jpg" alt="Impact Evaluation in Practice" title="Impact Evaluation in Practice" width="100%">
</figure>
]

.pull-middle-3.center[
<figure>
 <img src="img/01/mastering-metrics.jpg" alt="Mastering Metrics" title="Mastering Metrics" width="90%">
</figure>
]

.pull-right-3.center[
<figure>
 <img src="img/01/mixtape.jpg" alt="Causal Inference: The Mixtape" title="Causal Inference: The Mixtape" width="100%" style="border: 1px solid #EE6A24">
</figure>
]

---

---

---

# Class technology

<figure>
 <img src="img/01/rstudio-logo.png" alt="RStudio logo" title="RStudio logo" width="50%">
</figure>
]

---

# The tidyverse

.center[
<figure>
 <img src="img/01/tidyverse.png" alt="The tidyverse" title="The tidyverse" width="50%">
</figure>
]

---

# The tidyverse

???

From "Master the Tidyverse" by RStudio

---

# R code, but reads like English!

```r
strike_damages_month <- bird_strikes %>% 
 group_by(Month) %>% 
 summarize(total_damages = sum(Cost, na.rm = TRUE),
 average_damages = mean(Cost, na.rm = TRUE))

ggplot(data = strike_damages_month, 
       mapping = aes(x = Month, y = total_damages)) + 
  geom_col() + 
  scale_y_continuous(labels = dollar) +
  labs(x = "Month",
       y = "Total damages",
       title = "Really expensive collisions happen in the fall?",
       subtitle = "Don't fly in August or October?",
       caption = "Source: FAA Wildlife Strike Database")
```
]
---

# Sucking

.center[
.box-inv-6.medium[“There is no way of knowing nothing about a subject to knowing something about a subject without going through a period of much frustration and suckiness.”]

.box-inv-6.medium[**“Push through. You’ll suck less.”**]

.box-inv-6.tiny[Hadley Wickham, author of **ggplot2**]
]

???

[Source for quote](https://twitter.com/Akbaritabar/status/1022057084802748416)

---

# Sucking

&nbsp;

.pull-left[
<figure>
 <img src="img/01/sucking-first-step.jpg" alt="Sucking at something is the first step towards being sort of good at something" title="Sucking at something is the first step towards being sort of good at something" width="100%">
</figure>
]

.pull-right[
<figure>
 <img src="img/01/sucking-nyt.png" alt="It's great to suck at something" title="It's great to suck at something" width="100%">
 <figcaption><a href="https://www.nytimes.com/2017/04/28/opinion/its-great-to-suck-at-surfing.html" target="_blank">Karen Rinaldi, "(It's Great to) Suck at Something"</a></figcaption>
</figure>
]

---

# Am I making you computer scientists?

.box-inv-6.huge[**No!**]

???

Words of wisdom from @allison_horst to her data analysis class this quarter: You don't need to be a mechanic to drive a car safely, and you don't need to be a computer scientist or developer to use R safely. #rstats #tidytuesday

https://twitter.com/ameliaritger/status/1214682596182904832?s=12

I always teach my students: there’s three levels of skill: driver, mechanic, engineer. You’re here to get your drivers license. But being able to change a tire can be helpful. Don’t worry not every one has to be an engineer. (I talk about R in social sciences)

https://twitter.com/Sumidu/status/1214695065387438083

---

# Learning R

.center[
<figure>
 <img src="img/01/learning-r.png" alt="Learning R through googling" title="Learning R through googling" width="70%" style="border: 1px solid #EE6A24">
</figure>
]

???

https://twitter.com/kierisi/status/898534740051062785

---

.box-6.huge[You can do this.]

---

---

# Goals for the class

.box-inv-6.medium[~~Become an expert with R~~]

.box-inv-6.medium[Speak and do causation]

.box-inv-6.medium[Design rigorous evaluations]

.box-6.medium[Change the world with data]

---

# Prerequisites

.box-inv-6.medium[**Math skills**]

.box-inv-6.small.sp-after[Basic algebra]

.box-inv-6.medium[**Computer science skills**]

.box-inv-6.small.sp-after[None]

.box-inv-6.medium[**Statistical skills**]

.box-inv-6.small[Regression, differences in means, and statistical significance]