NAEP Technical DocumentationSchool Trimming

Students were assigned relatively large weights in many cases because of

underestimation of the number of eligible students in some schools, which leads to inappropriately low probabilities of selection for those schools, and
the presence of large schools—especially high schools—in PSUs with small selection probabilities.

In the latter cases, the maximum permissible within-school sampling rate (determined by the maximum sample size allowed per school; see Student Sample Selection) could be smaller than the desired overall within-PSU sampling rate for students. Large weights arose also because very small schools were sampled with low probabilities, coupled with high levels of nonresponse and the compounding of nonresponse adjustments at various levels.

Students with notably large weights have an unusually large effect on estimates such as weighted means. The variability in weights contributes to the variance of an overall estimate by an approximate factor
1 + CV ² where CV ² is the relative variance of the weights. An occasional unusually large weight will likely produce large sampling variances of the statistics of interest, especially when the large weights are associated with students with atypical performance characteristics.

To reduce the effect of large contributions to variance from a small set of sample schools, the weights of such schools were reduced, i.e., trimmed. The trimming procedure introduces a bias but should reduce the mean square error of sample estimates.

The trimming algorithm for school weights was identical to that used since 1996 and trimmed the weight of any school that contributed more than a specified proportion, to the estimated variance of the estimated number of students eligible for assessment.

Let

M	=	Number of schools in which a specified assessment was conducted
W_i	=	Weight assigned to school "i" (i.e., π_i^–1 × SUBADJ_i × FTADJ_i)
	=	Estimated number of grade-eligible students in school "i" (i.e., the sum of the within-school weights, adjusted for nonresponse, for the students assessed)
	=

A rough approximation to the unit variance of the is

A trimming method was used that reduced the weight W_i for a small number of schools so that no school makes a contribution to the sum shown above that is greater than a specified proportion θ. That is, for any school "j", the weight W_j, after all weights have been trimmed as required, satisfies the condition

The weight is not to be altered if , and the condition is equivalent to

The school-level trimming was done iteratively. The weight for each school that failed to satisfy the inequality was reduced to the value given by the right-hand side of the inequality by using the initial weights. The procedure was iterated using the weights as trimmed.

The value of θ to be used was chosen by judgment to provide negligible bias while substantially reducing variance. The chosen value of θ was 10/M.

The trimming procedure was done separately within each NAEP region. A total of five schools were trimmed for grade 4, with trimming factors TRIM_i of 0.75 to 0.937; a total of four schools for grade 8, with trimming factors TRIM_i of 0.882 to 0.959; and a total of two schools for grade 12, with trimming factors TRIM_i of 0.776 to 0.917.

The trimmed school weights were computed as π_i^–1 × SUBADJ_i × FTADJ_i × TRIM_i.

Last updated 26 August 2008 (FW)

Printer-friendly Version