NAEP Technical DocumentationNonresponse Bias Analyses for the 2011 Assessment

NCES statistical standards call for a nonresponse bias analysis to be conducted for a sample with a response rate below 85 percent at any stage of sampling. Weighted school response rates for the 2011 assessment indicated a need for school nonresponse bias analyses for private school samples in grades 4 and 8 (operational subjects), for public school samples for the Bureau of Indian Education (BIE) in grades 4 and 8, and for private school samples in grades 8 and 12 (computer-based writing). Student nonresponse bias analyses were necessary for students in Detroit for grade 8 public schools (mathematics). Additionally, a student nonresponse bias analysis was required to handle the special case of session nonresponse in the science sample in grade 8 Colorado public schools. Thus, six separate school-level analyses and two separate student-level analyses were conducted.

The procedures and results from these analyses are summarized briefly below. The analyses conducted consider only certain characteristics of schools and students. They do not directly consider the effects of the nonresponse on student achievement, the primary focus of NAEP. Thus, these analyses cannot be conclusive of either the existence or absence of nonresponse bias for student achievement. For more details, please see the NAEP 2011 NRBA report (818KB).

Each school-level analysis was conducted in three parts. The first part of the analysis looked for potential nonresponse bias that was introduced through school nonresponse. The second part of the analysis examined the remaining potential for nonresponse bias after accounting for the mitigating effects of substitution. The third part of the analysis examined the remaining potential for nonresponse bias after accounting for the mitigating effects of both school substitution and school-level nonresponse weight adjustments. The characteristics examined were Census region, reporting subgroup (private school type), urban-centric locale, and size of school (categorical).

Based on the school characteristics available, for the private school samples at grades 8 and 12, there does not appear to be evidence of substantial potential bias resulting from school substitution or school nonresponse. However, the analyses suggest that a potential for nonresponse bias remains for the grade 4 private school samples. This result is evidently related to the fact that, among non-Catholic schools, larger schools were less likely to respond. Thus, when making adjustments to address the underrepresentation of non-Catholic schools among the respondents, the result is to overrepresent smaller schools at the expense of larger ones. The limited school sample sizes involved means that it is not possible to make adjustments that account fully for all school characteristics. Please see the full report for more details.

Each student-level analysis was conducted in two parts. The first part of the analysis examined the potential for nonresponse bias that was introduced through student nonresponse. The second part of the analysis examined the potential for bias after accounting for the effects of nonresponse weight adjustments. The characteristics examined were gender, race/ethnicity, relative age, National School Lunch Program eligibility, student disability (SD) status, and English language learner (ELL) status. For Colorado, additional school characteristics were examined: Census region, urban-centric locale, size of school (categorical), and state-based achievement (categorical).

Based on the student characteristics available, for the grade 8 Detroit student samples, there does not appear to be evidence of substantial potential bias resulting from student nonresponse. The same result can be concluded for grade 8 Colorado student samples, when considering student characteristics. However, analyses of the school characteristics suggest that a potential for nonresponse bias remains. Please see the full report for more details.

Last updated 24 September 2014 (JS)

Printer-friendly Version