Research Methods: Summarize Numerical Data

Political Research Quarterly http://prq.sagepub.com Improving Causal Inference: Strengths and Limitations of Natural Experiments Thad Dunning Political Research Quarterly 2008; 61; 282 originally published online Oct 3, 2007; DOI: 10.1177/1065912907306470 The online version of this article can be found at: http://prq.sagepub.com/cgi/content/abstract/61/2/282 Published by: http://www.sagepublications.com On behalf of: Western Political Science Association The University of Utah Additional services and information for Political Research Quarterly can be found at: Email Alerts: http://prq.sagepub.com/cgi/alerts Subscriptions: http://prq.sagepub.com/subscriptions Reprints: http://www.sagepub.com/journalsReprints.nav Permissions: http://www.sagepub.com/journalsPermissions.nav Citations http://prq.sagepub.com/cgi/content/refs/61/2/282 Downloaded from http://prq.sagepub.com at Yale University Library on March 1, 2010 Improving Causal Inference Strengths and Limitations of Natural Experiments Political Research Quarterly Volume 61 Number 2 June 2008 282-293 © 2008 University of Utah 10.1177/1065912907306470 http://prq.sagepub.com hosted at http://online.sagepub.com Thad Dunning Yale University, New Haven, Connecticut Social scientists increasingly exploit natural experiments in their research. This article surveys recent applications in political science, with the goal of illustrating the inferential advantages provided by this research design. When treatment assignment is less than “as if” random, studies may be something less than natural experiments, and familiar threats to valid causal inference in observational settings can arise. The author proposes a continuum of plausibility for natural experiments, defined by the extent to which treatment assignment is plausibly “as if” random, and locates several leading studies along this continuum. Research Methods: Summarize Numerical Data

Keywords: natural experiment; “as if” random; exogenous variation; continuum of plausibility; matching If I had any desire to lead a life of indolent ease, I would wish to be an identical twin, separated at birth from my brother and raised in a different social class. We could hire ourselves out to a host of social scientists and practically name our fee. For we would be exceedingly rare representatives of the only really adequate natural experiment for separating genetic from environmental effects in humans—genetically identical individuals raised in disparate environments. —Stephen Jay Gould (1996, 264) 1. Introduction Social scientists are increasingly exploiting natural experiments in their research. A recent search on “natural experiment” using “Google Scholar” (scholar .google.com) turned up more than 1 million hits; the results appearing on the first dozen pages suggest that economics and epidemiology are the leading fields to use the term, but political science is also well represented. An impressive volume of unpublished, forthcoming, and recently published studies in political science suggests the growing influence of the natural experimental approach. Table 1 provides a nonexhaustive list of several recent studies. As the name suggests, natural experiments take their inspiration from the experimental approach. A randomized controlled experiment (Freedman, Pisani, and Purves 1997, 4-8) has three hallmarks. First, the 282 response of experimental subjects to a “treatment” (or a series of treatments) is compared to the response of other subjects to a “control” regime, often defined as the absence of a treatment. Second, the assignment of subjects to treatment and control groups is done at random. Third, the application or manipulation of the treatment is under the control of the experimental researcher. Each of these traits plays a critical role in the experimental model of causal inference. Research Methods: Summarize Numerical Data

For example, in a medical trial of a new drug, the fact that subjects in the treatment group take the drug, while those in the control group do not, allows for a comparison of health outcomes across the two groups. Random assignment ensures that any difference in average outcomes between the two groups is not due to confounders, or factors other than the treatment that vary across the two groups and that may explain differences in health outcomes. Finally, experimental manipulation of the treatment establishes evidence for a causal relationship between the treatment and the health outcomes.1 Unlike true experiments, the data used in natural experiments come from naturally occurring phenomena— actually, in the social sciences, from phenomena that are often the product of social and political forces. Because the manipulation of treatment variables is not Author’s Note: I am grateful to Jake Bowers, Henry Brady, Bear Braumoeller, David Collier, David Freedman, Alan Gerber, Don Green, Susan Hyde, Ken Scheve, Jason Seawright, and three reviewers for their comments and suggestions. An earlier version of this article was presented at the annual meetings of the American Political Science Association, August 31–September 3, 2005. Downloaded from http://prq.sagepub.com at Yale University Library on March 1, 2010 Dunning / Improving Causal Inference 283 Table 1 Recent Natural Experiments in Political Science Study Substantive Focus Source of Alleged Natural Experiment Ansolabehere, Snyder, and Stewart (2000) Brady and McNulty (2004) The personal vote and incumbency advantage Voter turnout Electoral redistricting Cox, Rosenbluth, and Thies (2000) Incentives of Japanese politicians to joint factions Doherty, Green, and Gerber (2005) Effect of affluence on political attitudes Glazer and Robbins (1985) Congressional responsiveness to constituencies Grofman, Brunell, and Koetzle (1998) Midterm losses in the House and Senate Grofman, Griffin, and Berry (1995) Hyde (2006) Krasno and Green (2005) Miguel (2004) Miguel, Satyanath, and Sergenti (2004) Posner (2004) Stasavage (2003) Congressional responsiveness to constituencies The effects of international election monitoring on electoral fraud Effect of televised presidential campaign ads on voter turnout Nation building and public goods provision Economic growth and civil conflict Political salience of cultural cleavages Bureaucratic delegation, transparency, and accountability Precinct consolidation in California gubernatorial recall election Cross-sectional and temporal variation in institutional rules in two houses of Japanese parliament Random assignment of level of lottery winnings to lottery winners Electoral redistricting Party control of White House in previous elections House members who move to the Senate “As if” random assignment of election monitors to polling stations in Armenia Geographic spillover of campaign ads in states with competitive elections to some but not all areas of neighboring states Political border between Kenya and Tanzania Shocks to economic performance caused by weather Political border between Zambia and Malawi Variation in central banking institutions Research Methods: Summarize Numerical Data

Note: This nonexhaustive list includes published and unpublished studies in political science that either lay explicit claim to having exploited a “natural experiment” or that in my view adopt core elements of the approach. The published studies are largely those that turned up in searches of JSTOR and other electronic sources, while unpublished and forthcoming studies were either previously known to me or were pointed out to me by other scholars. generally under the control of the analyst, natural experiments are, in fact, observational studies. However, unlike other nonexperimental approaches, a researcher exploiting a natural experiment can make a credible claim that the assignment of the nonexperimental subjects to treatment and control conditions is “as if” random. Outcomes are compared across treatment and control groups, and both a priori reasoning and empirical evidence are used to validate the assertion of randomization. Thus, random or “as if” random of assignment to treatment and control conditions constitutes the defining feature of a natural experiment. Natural experiments can sometimes provide social scientists with an important means of improving the validity of their empirical inferences. As the examples discussed below will illustrate, natural experiments can be useful to political scientists investigating a wide range of topics; and although their use is becoming more common, many more natural experiments than we now realize may be available to researchers. In addition, natural experiments often take place at the intersection of quantitative and qualitative methods (Brady and Collier 2004). While the analysis of natural experiments is sometimes facilitated by the use of statistical and quantitative techniques, the detailed case-based knowledge often associated with qualitative research is crucial both to recognizing the existence of a natural experiment and to gathering the kinds of evidence that make the assertion of “as if” random assignment compelling. For these reasons, a detailed examination of the logic of natural experiments and a discussion of concrete applications should be of interest to a variety of scholars. The goal of this article is therefore to survey the use of natural experiments, particularly in political science, with an eye both to describing their powerful inferential logic and also to delineating the sorts of issues over which natural experiments may offer less leverage. After introducing and discussing several examples below, I make several general points about this increasingly common research design. 2. Natural Experiments: The Role of “As If” Randomization A first example comes from a domain far from the concerns of contemporary political science, but it Downloaded from http://prq.sagepub.com at Yale University Library on March 1, 2010 284 Political Research Quarterly nicely illuminates core features of a successful natural experiment. Nineteenth-century London suffered a number of devastating cholera outbreaks. John Snow, an anesthesiologist who first became interested in the causes of cholera transmission around 1848 (Richardson 1887/1936, xxxiv), conducted justifiably famous studies of the disease (Freedman 1991, 1999, 2005). At the time of Snow’s research, a variety of theories existed to explain cholera’s transmission, including the theory of bad air (miasma). Research Methods: Summarize Numerical Data

Snow’s experience as a clinician and his studies of the pathology of cholera deaths during previous epidemics, however, suggested that cholera might instead be an infectious disease carried through the water. Although various “causal process observations” (Collier, Brady, and Seawright 2004) supplied crucial support for the plausibility of Snow’s hypothesis, his strongest piece of evidence came from a natural experiment which he exploited during the epidemic of 1853 to 1854. Large areas of London were served by two water companies, the Lambeth company and the Southwark and Vauxhall company. In 1852, the Lambeth company moved its intake pipe further upstream on the Thames, thereby “obtaining a supply of water quite free from the sewage of London,” while the Southwark and Vauxhall company left its intake pipe in place (Snow 1855, 68). This move of the Lambeth water pipe provided Snow with his natural experiment. He obtained records on cholera deaths throughout London and also gathered information on the company that had provided water to the house of each deceased as well as the total number of houses served by each company in each district of the city. Snow then compiled a simple cross-tab showing the cholera death rate in households during the epidemic of 1853 to 1854, by source of water supply. Among houses served by Southwark and Vauxhall, the death rate from cholera was 315 per 10,000; among those served by Lambeth, it was a mere 37 (Snow 1855, Table IX, 86; see Freedman 2005).2 This dramatic difference between the two groups of houses suggested a large treatment effect—and compelling evidence for the impact of water supply source on deaths from cholera. Why did the move of the Lambeth water pipe constitute the basis of a credible natural experiment? In a natural experiment, assignment to treatment and control conditions—here, the water supply source—must be “as if” random. This implies that the water supply source is independent of observable and unobservable factors that might influence cholera death rates, and people do not move in response to treatment. At least as a necessary if not sufficient condition, the treatment and control groups are balanced with respect to other (measurable) variables that might explain cholera deaths. Snow presented various sorts of evidence to establish this “pretreatment equivalence” between the groups. His own words may be most eloquent: The mixing of the (water) supply is of the most intimate kind. The pipes of each Company go down all the streets, and into nearly all the courts and alleys. A few houses are supplied by one Company and a few by the other, according to the decision of the owner or occupier at that time when the Water Companies were in active competition. In many cases a single house has a supply different from that on either side. Each company supplies both rich and poor, both large houses and small; there is no difference either in the condition or occupation of the persons receiving the water of the different Companies. . . . It is obvious that no experiment could have been devised which would more thoroughly test the effect of water supply on the progress of cholera than this. Research Methods: Summarize Numerical Data

(Snow 1855, 74-75) Particularly important for Snow was the fact that residents did not appear to “self-select” into their source of water supply in ways that might be associated with the propensity to contract cholera. Absentee landlords often took the decision regarding which of the competing water companies would be chosen for a particular address; moreover, the decision of the Lambeth company to move its intake pipe upstream on the Thames was taken before the cholera outbreak of 1853 to 1854, and existing scientific knowledge did not clearly link water source to cholera risk. As Snow put it, the move of the Lambeth company’s water pipe meant that more than three hundred thousand people of all ages and social strata were divided into two groups without their choice, and, in most cases, without their knowledge [italics added]; one group being supplied with water containing the sewage of London, and, amongst it, whatever might have come from the cholera patients, the other group having water quite free from such impurity. (Snow 1855, 75) Snow’s investigation of cholera transmission provides several useful lessons about the elements of a convincing natural experiment (Freedman 1991, 1999). Snow went to great lengths to gather evidence and to use a priori reasoning to argue that only the water supply distinguished houses in the treatment Downloaded from http://prq.sagepub.com at Yale University Library on March 1, 2010 Dunning / Improving Causal Inference 285 group from those in the control group and, thus, the impressive difference in death rates from cholera was due to the effect of the water supply. Of course, to the extent that the “as if” random assignment fails, Snow’s study would be less useful as a way of making valid inferences about the sources of cholera transmission; yet the strength of the evidence (and subsequent medical research) bear out Snow’s conclusions. It is also worth noting that while the natural experiment may have been the coup de grace in a painstaking investigation into the causes of cholera transmission, Snow’s use of this natural experiment was complemented and indeed motivated by the other evidence that he had compiled. This body of evidence grew from Snow’s detailed knowledge of the progress of previous cholera outbreaks in England, his ability to cull information from a variety of sources, and especially his willingness to do on-the-ground “process tracing” and close-range exploration of seemingly disconfirming cases.3 This kind of close-range research also gave him the information he needed to discover and exploit his natural experiment, while his sense of good research design led him to recognize the inferential power of the natural-experimental approach. Snow used quantitative techniques such as two-by-two tables and cross-tabs that today may seem old-fashioned, but as Freedman (1999, 5) put it, “It is the design of the study and the magnitude of the effect that compel conviction, not the elaboration of technique.” Social-Scientific Examples Snow’s study of cholera provides an early example of a natural experiment and underscores core elements of a successful application of this research design. Other phenomena can also provide the basis for credible natural experiments, however—and may provide insight into substantive questions of greater concern to social scientists. Research Methods: Summarize Numerical Data

In one important class of natural experiments, researchers can take advantage of an actual randomizing device with a known probability distribution that assigns subjects to the treatment and control conditions. The most frequent example may be natural experiments that exploit prize lotteries. In a recent paper, for example, Doherty, Green, and Gerber (2006) were interested in assessing the relationship between income and political attitudes. They surveyed 342 people who had won a lottery in an Eastern state between 1983 and 2000 and asked a variety of questions about estate taxes, government redistribution, and social and economic policies more generally. Comparing the political attitudes of lottery winners to those of the general public (especially, those who do not play the lottery) is clearly a nonexperimental comparison, since people self-select as lottery players, and those who choose to play lotteries may be quite different from those who do not, in ways that may matter for political attitudes. However, levels of lottery winnings are randomly assigned.4 Thus, abstracting from sample nonresponse and other issues that might threaten the internal validity of their inferences, Doherty, Green, and Gerber could obtain a clean estimate of the relationship between levels of lottery winnings and political attitudes.5 The example may demonstrate the power of natural experiments to rule out alternative interpretations of the findings—in the case of Doherty, Green, and Gerber’s (2006) study, the finding that lottery winnings affect attitudes toward the estate tax and perhaps some more narrow redistributive issues but not broader political and social attitudes. This is because unmeasured factors that might affect political attitudes should be statistically independent of the level of lottery winnings: just as in a true experiment, randomization takes care of the confounders.6 It is useful to note that in this class of natural experiment, unlike Snow’s, researchers do not need to depend on a priori reasoning or empirical evidence to defend the assumption of “as if” random assignment of subjects to treatment and control conditions: they simply exploit the true randomization afforded by the lottery. To readers in some fields, the idea of taking advantage of a true randomizing device to study the social world may seem far-fetched. How often will interesting substantive problems yield themselves to the kind of actual randomization that Doherty, Green, and Gerber (2006) could exploit? In fact, a number of studies in economics and political science have been able to make interesting use of various kinds of random mechanisms with known probability distributions. Researchers have exploited prize lotteries to study the effects of income on health (Lindahl 2002), happiness (Brickman, Janoff-Bulman, and Coates 1978; Gardner and Oswald 2001), and consumer behavior (Imbens, Rubin, and Sa …