Coverage Technical Report, Census of Population, 2016
10. Evaluation of coverage studies

10.1 Reverse Record Check

10.1.1 Introduction

The results of the largest coverage study, the Reverse Record Check (RRC), can be assessed by comparing its estimates with data on the same characteristics from other sources, such as the 2016 Census database and administrative data used by the Demographic Estimates Program (DEP). The purpose of making comparisons with RRC estimates is to evaluate the RRC estimates and to quantify conceptual and measurement differences.

Despite some conceptual differences between the RRC and the 2016 Census, the RRC estimates of persons enumerated in the 2016 Census can be compared with the census counts. To make the two numbers comparable, certain adjustments were first made to the census counts.

Estimates of the components of intercensal growth can be compared with estimates from other sources. The RRC estimates of the number of persons who died between the 2011 Census and the 2016 Census can be compared with the counts from vital statistics files. Estimates of net interprovincial migration calculated by the DEP based on Canada Revenue Agency data can be compared with RRC estimates. Lastly, RRC estimates of population growth components can be compared with similar estimates from administrative data.

10.1.2 Comparisons with census counts

Since the RRC’s single-stage stratified sampling design produces unbiased estimators, differences between RRC estimates and census counts are mainly due to sampling error in the RRC estimates, conceptual differences between the two sources, or systematic biases having impacts on the two sources, which result in an underestimate or overestimate of the characteristic being studied.

Enumerated persons

Provincial and national comparisons are presented in Table 10.1.2.1, along with the standard error of the RRC estimate and the t-value used to test the hypothesis that there is no difference between the RRC estimate and the comparable census count. The adjustments below were made in the published census counts to account for conceptual differences between the two sources:

• Adjustments based on the Dwelling Classification Survey were excluded because, while they were included in the census counts, they were not part of the RRC estimate of enumerated persons.
• The 2016 Census overcoverage estimate was subtracted because the census database contained overcovered persons, whereas the RRC estimate was based on the number of unique persons enumerated (and not on the number of enumerations).
• The estimate of the number of persons living outside Canada five years earlier (excluding intercensal immigrants and non-permanent residents) from the 2016 Census long-form questionnaire was also subtracted because the RRC estimates did not include the majority of these persons. For the same reasons, the estimated number of children aged 0 to 4 years who were born outside Canada but had Canadian citizenship was also subtracted.
• Similarly, for the provinces, the number of persons living in a territory five years earlier was subtracted because they were not covered by the RRC provincial sampling frames.
• The number of returning persons from reserves (who participated in the 2016 Census, but not in the 2011 Census) was also subtracted because the RRC estimates did not include the majority of these persons.

Nationally, the RRC estimate of the number of persons enumerated in the 2016 Census was slightly lower than the comparable census count (-0.09%). For the 1996 to 2011 censuses, the RRC estimates were between -0.08% and 0.12%. At the provincial level, the biggest differences were observed for Prince Edward Island and British Columbia; the estimate of the number of persons enumerated as part of the RRC underestimated the comparable census counts by 2,498 and 51,967 persons respectively. These differences were statistically significant. In the other provinces, the differences were not statistically significant. In previous cycles, significant differences were also observed. The most significant differences were investigated to make sure that there was no bias in the RRC classification (including, for example, province of residence on Census Day). Other factors may also play an important role in the observed differences. Apart from sampling error, biases in the adjustments (e.g., returning Canadians) applied to the published census counts to obtain conceptually comparable figures may be responsible for the differences. RRC non-response bias may also have played a role since the non-response adjustment was designed to obtain the best result for estimating missed persons rather than enumerated persons. Regular checks and quality controls were performed for all steps in the RRC. In view of the more significant differences for Prince Edward Island and British Columbia, a more detailed investigation was conducted to ensure that the operations and estimates were not affected by any of the above-mentioned errors or problems. No such errors or problems were detected.

10.1.3 Comparison with demographic estimates

Deceased persons

Table 10.1.3.1a provides a comparison of the estimated number of persons who died during the intercensal period (May 10, 2011, to May 9, 2016) by RRC province of classification with counts from vital statistics files. The RRC estimate excludes persons who died outside Canada when the country of death is known. At the national level, the RRC estimate exceeded the vital statistics count by 52,323 persons (4.1%), and this difference was statistically significant (t-value of 2.83). At the provincial level, the greatest differences were noted in Quebec (19,655, or 6.3%) and Ontario (30,215, or 6.4%), and these differences were statistically significant (t-value of 2.20 and 2.10 respectively). In the other provinces, the relative differences were between -4.0% and 4.8%, and they were not statistically significant.

Certain reasons may explain these significant differences. Firstly, the RRC estimate may include deaths that occur abroad, which are not included in vital statistics. In the RRC, if the country of death is known and is abroad, then the death is not included in the comparison of deceased persons in Table 10.1.3.1a. However, if the person is not found in the vital statistics files and the country of death is unknown, then they would be filed by default in their most recent province of residence in Canada. This situation occurs notably for persons only presumed to be deceased in the tax data. The deceased stratum from the 2011 Census frame contains many of these persons (approximately 20,000). Some of them probably died in Canada and could perhaps be found in the vital statistics files through more detailed manual searches. However, some of these persons probably died outside the country and do not appear in the vital statistics files. Table 10.1.3.1b provides a comparison of the RRC estimate of the number of persons who died during the intercensal period (May 10, 2011, to May 9, 2016) by province of death indicated in the vital statistics files (therefore, only for persons found in these files) with vital statistics counts. The differences that had been significant (Canada, Quebec and Ontario) were no longer significant, with t-values close to 1.0. However, the difference became significant in New Brunswick (-2,031, and a t-value of -2.50). Even if these last results do not seem indicative of issues related to the RRC estimates of the number of deceased persons, a more detailed investigation was conducted to confirm that no classification or other error was involved in the operations or estimates. No such errors or problems were detected.

Interprovincial migration

Table 10.1.3.2 compares RRC estimates of net interprovincial migration for the intercensal period with corresponding figures calculated by the DEP based on Canada Revenue Agency (CRA) files. In general, data on interprovincial migrants were not comparable because the RRC only took into account migration flows that occurred between the sampling frame reference date (e.g., May 10, 2011, for the census frame) and Census Day in 2016, whereas the DEP estimates took annual migration into account. For this reason, only net interprovincial migration estimates are presented. Also, the stratification of the 2011 Census frame using the most recent province of residence based on tax data produced more accurate provincial estimates of persons missed by the 2016 Census, but made it more difficult to estimate through the RRC the net interprovincial migration for the sample from this census frame.

The only observed difference that was statistically significant was in Manitoba (t-value of -2.18), where the RRC estimate of net migration was significantly more negative than the DEP estimate. Both sources estimated a negative net migration, but the size differed by source. This difference could be explained by the sampling error from the 2011 Census sample, and by a larger number of recent immigrants who left the province according to the RRC. For Prince Edward Island, Quebec and Ontario, the differences were high, but not significant. As was the case for Manitoba, the RRC net migration estimate was significantly more negative than the DEP estimate for Prince Edward Island and Quebec. The analyses for these two provinces indicate that this difference was mainly caused by migrations of newly established immigrants that were not entirely captured in the tax data used by the DEP. In Ontario, the RRC estimated a slightly positive net migration, which was contrary to the negative DEP estimate. This difference could reflect the situation observed in several other provinces. The RRC could indicate interprovincial migrations of certain recent immigrants that were not measured by the DEP. In Newfoundland and Labrador, the RRC net migration was negative, but it was positive according to the CRA. However, the t-value was below 1. In the five other provinces, the RRC and CRA estimates were similar and went in the same direction.

10.1.4 Components of population growth

The Demography Division conducted an extensive comparison of RRC estimates of the intercensal population growth components with demographic estimates derived from administrative data (this topic is also discussed in Section 10.3). The RRC estimates of the demographic components are a by-product of the RRC and therefore are not necessarily precise. However, these data provide information on population growth components, which could potentially be more related to measurement error from the DEP.

Total population growth estimates from these two sources are presented in Table 10.1.4. The estimates of returning Canadians and persons living on Indian reserves or in Indian settlements that were incompletely enumerated in 2011 and enumerated in 2016 were added to the RRC estimates to make them comparable with the DEP estimates. The DEP estimates come from a combination of several population growth components, such as births, deaths and immigration, which were subject to varying amounts of measurement error depending on the source.

At the national level, the RRC estimate is lower by 126,121 (or 6.8%) than the DEP estimate. At the provincial level, the greatest differences are noted for Quebec (-79,610) and British Columbia (34,258).

10.2 Census Overcoverage Study

Many changes were made to the 2016 COS methodology to identify more overcoverage cases than in 2011. The evaluation to gauge the success of the 2016 COS had two objectives: to measure overcoverage not detected by the COS, and to quantify the improvement attributable to the methodological changes made since 2011. The Automated Match Study (AMS) is a useful tool to achieve both objectives since its methodology has remained essentially unchanged since 2001. It is particularly useful for addressing the significant problem of breaking down any increase in the estimated overcoverage into two components: higher overcoverage in the studied population, and additional overcoverage detected because of improvement in the COS methodology.

10.2.1 Comparison of the 2011 and 2016 AMSs

The 2016 AMS was carried out using the same methodology as the 2011 AMS, and then the two studies were compared. This made it possible to estimate the relative differences in overcoverage at several levels (e.g., national, provincial and territorial) between 2011 and 2016. The results of the comparison are shown in Table 10.2.1.

The 2016 AMS revealed that overcoverage had once again increased, as had been the case between each of the previous censuses. The 2016 AMS indicated an 8.0% increase in the estimated number of overcovered persons compared with the 2011 AMS. This is consistent with the estimate produced by the COS, which indicated a relative increase of 11.8%. The observed increase was closer across studies this time, compared with the increase observed between 2006 and 2011.

At the provincial and territorial level, the variation between estimates from the 2011 AMS and the 2016 AMS was positive for 8 of the 13 provinces and territories. The variation in estimated overcoverage between the 2011 and 2016 censuses was consistent for the AMS and COS for all provinces and territories except three—Prince Edward Island, Nova Scotia and New Brunswick—where the AMS estimate indicated a decrease in overcoverage and the COS estimate indicated an increase. From 2006 to 2011, the variation moved in opposite directions in two places, Prince Edward Island and Manitoba.

10.2.2 Comparison of the 2016 COS and the 2016 AMS

The results of the 2016 COS were compared with the results of the 2016 AMS to estimate overcoverage missed by the COS but detected by the AMS, overcoverage missed by the AMS but detected by the COS, and overcoverage identified by both studies. These kinds of differences are to be expected because of the different approaches taken in the COS (person-based) and the AMS (household-based). The comparison was carried out in two steps.

The first step was to estimate the overcoverage detected by both the AMS and the COS in the COS sampling frames, i.e., overcoverage in the AMS domain of the COS. This overcoverage was estimated by matching person pairs that were in the AMS sampling frame with duplicates in the COS sample. It was estimated using the COS sample.

The second step was to estimate overcoverage detected by the AMS, but not by the COS. This overcoverage was equal to the total overcoverage for all AMS household pairs that contained no COS person pairs. It was estimated by matching the COS person pairs with the duplicates in the AMS sample. Unmatched AMS duplicates were the portion not detected by the COS.

The results of comparing the COS with the AMS are presented in Table 10.2.2.

The left side of Table 10.2.2 contains the national estimates based on the COS sample:

• overcoverage in the COS frame: 695,828
• overcoverage in the COS initial frame and the AMS frame: 433,140, or 62.2% of the total overcoverage detected using the COS
• overcoverage in the COS frame but not in the AMS frame: 262,688, or 37.8% of the total overcoverage detected using the COS.

The right side contains the following national estimates based on the AMS sample:

• overcoverage in the AMS frame: 464,993
• overcoverage in the COS frame and the AMS frame: 453,484, or 97.5% of the total overcoverage detected using the AMS
• overcoverage in the AMS frame but not in the COS frame: 11,508, or 2.5% of the total overcoverage detected using the AMS.

As shown in Table 10.2.2, the COS and AMS can both be used to estimate the overcoverage covered by the two studies. For this common portion, the AMS estimate exceeds the COS estimate by 20,344. The two estimates are consistent with each other, and their difference is not statistically significant at the 95% confidence level.

10.3 Demographic estimates

10.3.1 Error of closure

Statistics Canada’s DEP determines provincial and territorial population counts on Census Day by summing census population counts, estimates of census net undercoverage (CNU) and the population estimate for incompletely enumerated Indian reserves. The DEP then extends these adjusted counts to July 1, and they become the base for postcensal demographic estimates.

When determining these adjusted counts, the DEP evaluates the quality of the postcensal estimates that it produced in the five-year period preceding the census. The evaluation focuses on the difference between the postcensal estimates for Census Day and the adjusted population count for this census. This difference is referred to as the error of closure. The detailed examination of this error is the main quality measure of the postcensal estimates.

Table 10.3.1 shows the errors of closure for 2006, 2011 and 2016 by province and territory, and for Canada. Note that a positive error of closure means that the postcensal demographic estimate is higher than the adjusted census count. At the national level, the error of closure for 2016 was 110,310, for an error rate of 0.31%. The national demographic estimates therefore overestimated Canada’s population. The error and error rate in 2016 were lower than in 2011, but higher than in 2006.^{Note 1} Five provinces had errors of closure greater than 1% or less than -1% in 2016: Prince Edward Island (1.88%), Quebec (1.05%), Saskatchewan (1.06%), Alberta (1.05%) and British Columbia (-2.07%). By comparison, in 2011, four provinces and two territories had similar errors of closure. In 2016, seven provinces and one territory had larger errors of closure (in absolute value terms) than in 2011.

10.3.2 Accuracy of postcensal estimates

For the purposes of producing the DEP estimates, the census coverage studies are used to adjust census counts for CNU. However, since these studies are based in part on sample surveys, the CNU results contain some statistical variability attributable to sampling. To determine whether the errors of closure discussed above are statistically significant, the standard error of the adjusted census counts must be taken into account. Moreover, since the 2011 adjusted census counts were used as the base population for the 2011 to 2016 postcensal estimates, a standard error that combines the statistical variability of the adjusted census counts for 2011 and 2016 was calculated for Canada and for each province and territory.

Table 10.3.2 shows the 2016 error of closure for Canada, the provinces and territories; the combined standard error of the 2011 and 2016 adjusted census counts; and the t-value.^{Note 2} The error of closure is statistically significant at a 95% confidence level for Prince Edward Island, Quebec, Alberta and British Columbia. For these provinces, the variability attributable to sampling of the 2011 and 2016 adjusted census counts therefore does not explain the majority of the error of closure.

The components of population growth estimated by the DEP were compared with those from other sources, mainly the RRC, to determine the components that could be more closely linked to the error of closure. This analysis focused on the four provinces for which the error was statistically significant. Interprovincial migration, particularly that of recent immigrants, could explain part of the error of closure calculated for Prince Edward Island and Quebec. In addition to the variability in measuring the net coverage error, several components of population growth could help to explain the error calculated for Alberta and British Columbia. However, it is difficult to identify a primary factor. Lastly, emigration generally remains a demographic phenomenon that is particularly difficult to measure.

• Previous
• Next