Table of Contents | Search Technical Documentation | References

NAEP Analysis and Scaling → Scale Linking and Transformation to the Reporting Metric → NAEP Variance Estimation During Transition to Digitally Based Assessment

NAEP Technical DocumentationNAEP Variance Estimation During Transition to Digitally Based Assessment

Suppose common population linking is used to align the digitally based assessment (DBA) scores from future reporting samples to the existing trend scale used in reporting the paper-based assessment (PBA) results (referred to as trend PBA). In NAEP, two randomly equivalent samples from the same population, one administered with DBA, while the other one with PBA (referred to as bridge PBA), are used in estimating the common population linking function. An approach is described below for estimating error variance of the group statistics for the DBA assessments. The statistic of interest, t hat , can be a mean, a percentage, an achievement level percentage, etc.

Procedures for estimating error variance for DBA group scale score statistics, when comparing groups between DBA and trend PBA—External Linking

Suppose that the common population linking functions are estimated using two randomly equivalent samples—the DBA sample and the bridge PBA sample, and then applied to link the results of a third independent DBA sample to the existing trend scale in reporting the trend PBA results. This is referred to as external linking.

Suppose the DBA results are linked to the trend scale in reporting PBA via external linking. When comparing the DBA result, t hat sub D B A , to the trend PBA result, t hat sub P B A , where PBA and DBA are two independent samples, the error variance of the absolute value of, t hat sub D B A, end sub, minus t hat sub P B A, end absolute value is

the variance hat of, t hat sub P B A, plus the variance hat of, t hat sub D B A . (1)

The variance hat of, t hat sub P B A is estimated as

the variance hat of, t hat sub P B A, equals the sampling variance hat of, t hat sub P B A, , plus the measurement variance hat of, t hat sub P B A, , (2)

where the sampling variance hat of, t hat sub P B A and the measurement variance hat of, t hat sub P B A are the sampling and measurement variances for .

the variance hat of, t hat sub D B A is estimated as

the variance hat of, t hat sub D B A, equals the sampling variance hat of, t hat sub D B A, plus the measurement variance hat of, t hat sub D B A, plus the linking error variance hat of, t hat sub D B A , (3)

where the sampling variance hat of, t hat sub D B A and the measurement variance hat of, t hat sub D B A are the sampling and measurement variances for . The term the linking error variance hat of, t hat sub D B A ] is the linking error variance associated with the uncertainty in estimating the common population linking function. When external linking is used, is estimated using a quadratic function which describes the relationship between the linking variance and . Specifically, the quadratic function for any subgroup has the following general form:

the linking error variance hat of, t hat sub D B A, equals a, hat times t hat squared sub D B A, end sub, plus b hat times t hat sub D B A, end sub, plus c hat , (4)

where a, hat , b hat , and c hat are coefficients estimated through a Monte Carlo based method. See Mazzeo, Donoghue, Liu, and Xu (2018) for more discussion of the method of calculating the linking variance for external linking.

The approach described here is used in estimating the error variance of the statistics from Mathematics and Reading DBA at grades 4 and 8, when comparing the group scale score statistics from the DBA in 2017 and future years to those from PBA in 2015 and earlier years. Tables that provide the quadratic function coefficients for means and percentages, standard deviations, and achievement-level percentages can be accessed via the table below.

The linking error component derived from external linking does not apply when

estimating error variance for DBA group scale score statistics themselves (procedures for estimating group scale score statistics and their standard error),
comparing group scale score statistics from DBA, or
comparing group scale score statistics from PBA.

Links to quadratic function coefficients used for linking variance calculation, by subject: 2017
Year	Subject area
2017	Mathematics
2017	Reading
SOURCE: U.S. Department of Education, Institute of Education Sciences, National Center for Education Statistics, National Assessment of Educational Progress (NAEP), 2017 Assessments.

Last updated 14 June 2022 (SK)

Printer-friendly Version