8. Variance

The error in an estimate is the difference between the estimate and the actual value of what is being estimated. Sampling error and household non-response error are two sources of error in the NHS. Sampling error stems from the fact that the estimates are based on observations from a sample and not from the entire population. Household non-response error occurs when households selected in the sample do not respond to the survey. Estimation methods also have an impact on the error. Some methods are more precise than others in measuring a particular characteristic of the population.

The error has two components: a random component, the variance; and a systematic component, the bias. The variance measures how much the estimate varies from the average that would result from hypothetical repetitions of the survey process. The variance can be estimated using data from respondents in the sample. The bias is the difference between the average estimate that would result from hypothetical repetitions of the survey process and the actual value of the characteristic being estimated. Non-response bias is covered in Chapter 5. The sampling and estimation methods used in the NHS generate negligible bias. This chapter deals with sampling variance and variance due to household non-response.

To estimate the variance, the first step is to derive estimators using mathematical formulas. Those formulas have to take into account the various sampling and weighting steps described in previous chapters. The formulas specify how data from respondents are used to produce variance estimates. To perform the derivations, certain assumptions and simplifications had to be made. First, it was assumed that the sample and subsample selected by systematic stratified sampling were selected by stratified simple random sampling without replacement. This assumption is necessary when systematic sampling is performed. The assumption is reasonable because the strata are small and their population can therefore be treated as homogeneous. The second assumption concerns the mechanism that generates household non-response. It was assumed to be equivalent to a Poisson sampling process in which a household's probability of response/selection is equal to the ratio of its weight assigned before non-response adjustment weight transfer to the weight assigned to it after the transfer. For example, a respondent household whose weight doubled following the transfer would have an estimated response probability of 50%. It was also assumed that this probability was known, not estimated. Making these assumptions about the non-response mechanisms means that the non-response bias in the variance estimate is ignored. The last simplification is that surprise respondents were excluded from the variance estimation process. This makes the process much easier and has little impact on the estimate since surprise respondents make up a small fraction of all respondents. The technical details of the variance estimation process are provided in Verret (2013).

Since the variance estimators based on the formulas described above tend to underestimate the actual variance, the final step was to correct them using simulations. To that end, an artificial population was first generated using sample data. Multiple samples were then selected independently (i.e., replicates), and the weighting methods were applied to each one. The calculated variance estimators were compared with the actual variability from replicate to replicate. This provides an upward adjustment that is applied to all the variance estimates calculated with the formulas.

The estimated variance can be used to construct several types of measures of an estimate's variability. For example, it can be used to calculate standard errors or coefficients of variation (CVs). The standard error is equal to the square root of the variance. The CV is the ratio of the estimate's standard error to the estimate itself. The smaller these measures of variability are, the more precise the estimate. The CV is a particularly interesting measure of variability in that it does not depend on the unit of measure of the characteristic being studied and can be expressed as a percentage. It is important to carefully distinguish between these measures of variability and other measures of quality that are not, strictly speaking, measures of variability. Examples of such measures are the bias indicators, response rate and global non-response rate of the NHS. The response rate is an indicator of the risk associated with household non-response error. The global non-response rate is an indicator of the risk of error due to household non-response and item non-response. Table 8.1 shows the CVs of various estimates at the Canada, provincial and territorial levels. The contents of this table can be consulted or downloaded at the following address: http://www12.statcan.gc.ca/nhs-enm/2011/ref/cv/index.cfm?Lang=E.

The sampling and estimation methods used in the 2011 NHS are different from the ones used for the 2006 Census long questionnaire. The magnitude and effect of non-response are very different in 2011. All that has an impact on the variability of the estimates. A methodological note comparing the CVs of the 2011 estimates with the CVs of the 2006 estimates for some variables is available at http://www12.statcan.gc.ca/nhs-enm/2011/ref/cv/cvnote.cfm?Lang=E.

Table 8.1
Coefficients of variation (%) of the 2011 National Household Survey – Canada, provinces and territories