Incorporating interpreter variability into estimation of the total variance of land cover area estimates under simple random sampling

Area estimates of land cover and land cover change are often based on reference class labels determined by analysts interpreting satellite imagery and aerial photography. Different interpreters may assign different reference class labels to the same sample unit. This interpreter variability is typically not accounted for in variance estimators applied to area estimates of land cover. A simple measurement model provides the basis for an estimator of the total variance (V_Total) that takes into account both sampling variance and interpreter variance. This method requires two or more reference class interpretations (i.e., repeated measurements) obtained by analysts, working independently of each other, for the full sample or a random subsample of the full sample. Estimators of the total variance (V̂Total">V̂Total) and the variance component attributable to interpreters (V̂1">V̂1) were obtained for the case of two reference class interpretations per repeated sample unit. To evaluate the effect of interpreter variability on variance estimation, we used land cover reference data interpreted by seven analysts who each interpreted the same 300 sample pixels from a region of the Pacific Northwest of the United States. From these data, we estimated the contribution of interpreter variance to the total variance (i.e., V̂1/V̂Total">V̂1/V̂Total) and the relative bias of the standard simple random sampling variance estimator (V̂stand">V̂stand) as an estimator of V_Total, defined as 100%*(V̂stand−V̂Total">V̂stand−V̂Total)/V̂Total">V̂Total. For each of five land cover classes, we computed V̂1">V̂1, V̂Total">V̂Total, and V̂stand">V̂stand using the sample data from each of the 21 possible pairwise combinations of the seven interpreters, and then calculated the mean of V̂1/V̂Total">V̂1/V̂Total and the mean of the estimated relative bias of V̂stand">V̂stand over these 21 pairs. Based on the mean of V̂1/V̂Total">V̂1/V̂Total per class, interpreter variance contributed from 25% (cropland) to 76% (grass/shrub) of the total variance, indicating that interpreter variance was a non-negligible component of the total variance. Typically, the standard variance estimator, V̂stand">V̂stand, underestimated the total variance with the mean estimated relative bias ranging from −3% (cropland) to −33% (grass/shrub). Classes with greater inconsistency between pairs of interpreters had larger contributions of interpreter variance to the total variance (V̂1/V̂Total">V̂1/V̂Total) and larger negative estimated relative bias of V̂stand">V̂stand. Given that interpreter variance can contribute substantially to the total variance, the repeated measurements approach offers a practical way to incorporate this variability into an estimator of the total variance.