How (Not) to Estimate the Design Effect of a Complex Sampling Scheme: A Case Study of the Polish Section of the European Social Survey, Round 5

Abstract

Design effect (DEFF) is a measure used to assess the effectiveness of a particular sampling scheme. Even though its definition is remarkably simple (cf. Kish 1965: 258), its practical implementation turns out to be problematic. Researchers therefore usually simplify the estimation of DEFF by independently determining the values of three components, namely, the clustering effect (DEFFc), the stratification effect (DEFFs) and the effect of unequal sampling probabilities (DEFFp) and by multiplying these partial measures to obtain a measure of overall effect. However, the validity of such a simplified version depends on strict formal requirements which are met only in a few sampling schemes. The subject of the analysis presented here is the sampling scheme in the Polish section of round 5of the European Social Survey (ESS). It will be shown that the method of DEFF estimation applied by the Polish coordinators of the project, which is compatible with the methodological recommendations of ESS (cf. Lynn et al. 2007: 114),does not satisfy the formal criteria that would validate its use. The author proposes two other ways of estimating the size of DEFF (cf. Gabler et al. 2006: 116-117) appropriate for the sampling scheme in ESS5-PL. Empirical analyses indicate that the use of the simplified procedure of DEFF prediction leads to significant underestimation of variance inflation in the sample design of ESS5-PL and, in turn, to overestimation of effective sample size.