NAEP Technical DocumentationNonresponse Bias Analyses for the 2017 Assessment

NCES statistical standards call for a nonresponse bias analysis to be conducted on a sample with a response rate below 85 percent at any stage of sampling. Weighted school response rates for the 2017 assessment indicated a need for school-level nonresponse bias analyses for the mathematics and reading private school samples at grades 4 and 8. Based on weighted student nonresponse rates, nonresponse bias analyses at the student level were required for the grade 8 public schools in New York state in mathematics and reading and for the Washington, DC public school TUDA district in mathematics. Overall, five nonresponse bias analyses were conducted: two at the school level and three at the student level. Since mathematics and reading assessments were generally conducted in the same schools, it was not necessary to carry out separate school-level nonresponse bias analyses by subject.

The procedures and results from these analyses are summarized briefly below. The analyses conducted consider only certain characteristics of schools and students. They do not directly consider the effects of the nonresponse on student achievement, the primary focus of NAEP. Thus, these analyses cannot be conclusive of either the existence or absence of nonresponse bias for student achievement. For more details, please see the NAEP 2017 Nonresponse Bias Analysis Report.

School-level Nonresponse Bias Analyses

Each school-level analysis was conducted in three parts. The first part of the analysis looked for potential nonresponse bias that was introduced through school nonresponse. The second part examined the remaining potential for nonresponse bias after accounting for the effects of substitution. And the third part examined the remaining potential for nonresponse bias after accounting for the effects of both school substitution and school-level nonresponse weight adjustments. The characteristics examined were Census region, private school reporting group (Catholic/non-Catholic), urban-centric locale, school grade size category, and race/ethnicity percentages (private school samples only). In addition, two measures of the mean size of enrollment in the respective grades were considered: one is the mean grade enrollment size, i.e. mean size of school attended by an average student, which is estimated using the enrollment-size-adjusted school weight; and the other is mean-estimated grade enrollment, which is estimated using the school weight without the enrollment size adjustment.

The analyses for private schools at grades 4 and 8 show that substitution did not seem to have much effect in reducing nonresponse bias. Nonrespose adjustments did have a large impact in reducing the nonresponse bias; however, they did not eliminate this bias completely. After nonrespose adjustments, nonresponse bias still remained for urban-centric locale, percent White, and percent Black at grade 4. At grade 8, nonrespose bias remained only for mean enrollment size. Please see the full report for more details.

Student-level Nonresponse Bias Analyses

Each student-level analysis was conducted in two parts. The first part of the analysis examined the potential for nonresponse bias that was introduced through student nonresponse. The second part examined the potential for bias after accounting for the effects of nonresponse weighting adjustments. The characteristics examined were gender, race/ethnicity, relative age, National School Lunch Program eligibility, student disability (SD) status, and English learner (EL) status.

The analyses for the relevant public school samples at grades 4 and 8 show that after applying student nonresponse adjustments, there does not appear to be evidence of nonresponse bias. Please see the full reportfor more details.

Printer-friendly Version