NAEP Technical DocumentationStudent Weight Raking Adjustment for the 2017 Assessment

			Development of Final Raking Dimensions Raking Adjustment Control Totals Raking Adjustment Factor Calculation

Weighted estimates of population totals for student-level subgroups for a given grade will vary across subjects even though the student samples for each subject generally come from the same schools. These differences are the result of sampling error associated with the random assignment of subjects to students through a process known as spiraling. For state assessments in particular, any difference in demographic estimates between subjects, no matter how small, may raise concerns about data quality. To remove these random differences and potential data quality concerns, a step was added to the NAEP weighting procedure in 2009. This step adjusts the student weights in such a way that the weighted sums of population totals for specific student groups are the same across all subjects. It was implemented using a raking procedure and applied only to public school assessments.

Raking is a weighting procedure based on the iterative proportional fitting process developed by Deming and Stephan (1940) and involves simultaneous ratio adjustments to two or more marginal distributions of population totals. Each set of marginal population totals is known as a dimension, and each population total in a dimension is referred to as a control total. Raking is carried out in a sequence of adjustments. Sampling weights are adjusted to one marginal distribution and then to the second marginal distribution, and so on. One cycle of sequential adjustments to the marginal distributions is called an iteration. The procedure is repeated until convergence is achieved. The criterion for convergence can be specified either as the maximum number of iterations or an absolute difference (or relative absolute difference) from the marginal population totals. More discussion on raking can be found in Oh and Scheuren (1987).

For NAEP 2017, the student raking adjustment was carried out separately in each state and TUDA districts for the mathematics and reading public school samples at grades 4 and 8. The dimensions used in the raking process were race/ethnicity, SD/EL status, and gender. National School Lunch Program (NSLP) eligibility, which has been used as a raking dimension in NAEP years prior to 2015, was not used as a raking dimension in 2015 and 2017 because of the instability of these data in many states. In addition, raking was implemented for the national public school sample for writing at grades 4 and 8 at the national level. The dimensions used in the raking process for writing were the same as those for mathematics and reading (i.e., race/ethnicity, SD/EL status, and gender). The control totals for the raking dimensions for all three subject-based samples (i.e., mathematics, reading, and writing) for each grade (4 and 8) were obtained from summing the combined DBA/PBA student sample weights of the mathematics and reading public school samples combined.

Printer-friendly Version