Strong Control of the Family-wise Error Rate in Observational Studies
An effect modifier is a pretreatment covariate such that the magnitude of the treatment effect or its stability changes with the level of the covariate. Generally, other things being equal, larger treatment effects and less heterogeneous treatment effects are less sensitive to unmeasured biases in observational studies. It is known that when there is effect modification, an overall test that ignores an effect modifier may report greater sensitivity to unmeasured bias than a test that combines results at different levels of the effect modifier. This known combined test reports that there is evidence of an effect somewhere that is insensitive to bias of a certain magnitude, but it does not draw inferences about affected subgroups. If there is effect modification, one would like to identify specific subgroups for which there is evidence of effect that is insensitive to small or moderate biases. We propose an exploratory method for discovering effect modification combined with a confirmatory method of simultaneous inference that strongly controls the family-wise error rate in a sensitivity analysis, despite the fact that the groups being compared are defined empirically. A new form of matching, strength k matching, permits a search through more than k covariates for effect modifiers, yet no pairs are lost providing at most k covariates are selected to group the pairs. In a strength k match, each set of k covariates is exactly balanced, though a set of > k covariates may exhibit imbalance. We apply the method to study the effects of the powerful earthquake that struck Chile in 2010. This is joint work with Jesse Hsu, Jose Zubizarreta and Paul Rosenbaum.