Best Practices

We hope our study contributes to the credibility revolution, not by dissuading the use of natural experiments, but rather by helping researchers make better inferences when natural experiments are reused. On this page, we highlight best practices as discussed in our Journal of Finance article.

p-values

Our paper shows that when a setting is reused, p-values no longer have their usual interpretation. As such, we advocate the use of multiple testing corrections to adjust statistical significance for the number of outcomes examined.

However, while we advocate the use of multiple testing corrections, we stress that p-values are only one of many inputs that should be used in evaluating research and making decisions. We caution that multiple testing correction methods are not a panacea; simply clearing the hurdle of adjusted critical values does not mean that a research design is valid. A p-value of 0.09 should not be viewed as proving a result exists, nor should a p-value of 0.11 be viewed as proving there is no result. Rather, p-values are just one of many inputs that assist with inference, along with information about the proposed economic mechanism and the validity of the research design. As such, we recommend a number of other best practices.

Other Best Practices

1. As discussed in our paper, one way to improve inference is to find corroborating evidence that supports the main finding, either by finding new data or by testing additional hypotheses.

2. In addition, researchers should attempt to provide supporting evidence of causal channels. For example, in the case of the regulation SHO pilot program, the program was designed to remove a potential restriction to short-selling. If a researcher believes the removal of this restriction changed the information content of stock prices, they should first establish that the program changed trading behavior (in particular, trading by short sellers). See the discussion of causal chains and the "best foot forward" policy in our paper.

3. Finally, when reusing a setting, researchers should acknowledge findings from the existing literature that examines the setting. For example, Morck and Yeung (2011) note that "each successful use of an instrument creates an additional latent variable problem for all other uses of that instrument." This concern applies more generally within the context of all natural experiments, not just instrumental variables settings. Researchers reusing an experimental setting should reconcile their exclusion restrictions with all existing empirical evidence at the time their study is written.

For more on all these points, see the published version of the paper.