Experiments in producing nonresponse bias

Abstract

While nonresponse rates in household surveys are increasing in most industrialized nations, the increasing rates do not always produce nonresponse bias in survey estimates. The linkage between nonresponse rates and nonresponse bias arises from the presence of a covariance between response propensity and the survey variables of interest. To understand the covariance term, researchers must think about the common influences on response propensity and the survey variable. Three variables appear to be especially relevant in this regard: interest in the survey topic, reactions to the survey sponsor, and the use of incentives. A set of randomized experiments tests whether those likely to be interested in the stated survey topic participate at higher rates and whether nonresponse bias on estimates involving variables central to the survey topic is affected by this. The experiments also test whether incentives disproportionately increase the participation of those less interested in the topic. The experiments show mixed results in support of these key hypotheses.