NAEP Technical DocumentationNonresponse Bias Analyses for the 2015 Assessment

NCES statistical standards call for a nonresponse bias analysis to be conducted on a sample with a response rate below 85 percent at any stage of sampling. Weighted school response rates for the 2015 assessment indicated a need for school-level nonresponse bias analyses for private school samples at grades 4, 8, and 12 and for public school samples of the Bureau of Indian Education (BIE) at grades 4 and 8. Based on weighted student nonresponse rates, nonresponse bias analyses at the student level were required for the overall public and private school student samples at grade 12. Thus, five separate school-level analyses and two separate student-level analyses were conducted.

The procedures and results from these analyses are summarized briefly below. The analyses conducted consider only certain characteristics of schools and students. They do not directly consider the effects of the nonresponse on student achievement, the primary focus of NAEP. Thus, these analyses cannot be conclusive of either the existence or absence of nonresponse bias for student achievement. For more details, please see the NAEP 2015 Nonresponse Bias Analysis Report (525.5 KB).

Each school-level analysis was conducted in three parts. The first part of the analysis looked for potential nonresponse bias that was introduced through school nonresponse. The second part of the analysis examined the remaining potential for nonresponse bias after accounting for the effects of substitution. The third part of the analysis examined the remaining potential for nonresponse bias after accounting for the effects of both school substitution and school-level nonresponse weight adjustments. The characteristics examined were Census region, private school reporting group (Catholic/non-Catholic), urban-centric locale, school grade size category, and race/ethnicity percentages (private school samples only). In addition, two measures of the mean size of enrollment in the respective grades were considered: one is the mean grade enrollment size, i.e. mean size of school attended by an average student, which is estimated using the enrollment size adjusted school weight; and the other is mean estimated grade enrollment, which is estimated using the school weight without the enrollment size adjustment. Since practically every student in a BIE school is American Indian, race/ethnicity was not included in the analyses for the BIE samples.

Based on the school characteristics available for the private school samples at grade 12, there does not appear to be evidence of substantial potential bias after school substitution and school nonresponse adjustments. However, the analyses suggest that a potential for nonresponse bias remains for the grade 4 and 8 private school samples. The analysis suggested potential significant bias for the school grade size category for grades 4 and 8 private schools, and mean grade 8 enrollment size (where the mean is estimated regarding students as the units of analysis) for grade 8 private schools, after nonresponse adjustments.

Each of the grade 4 and 8 BIE school samples consisted of a census of schools, and so any differences between the responding and original samples are statistically significant. There was a slight decrease in nonresponse bias for the school grade size category, estimated grade enrollment, and the mean grade enrollment size (where the mean is estimated regarding students as the units of analysis) for both samples after the school nonresponse adjustments were applied. Please see the full report for more details.

Each student-level analysis was conducted in two parts. The first part of the analysis examined the potential for nonresponse bias that was introduced through student nonresponse. The second part of the analysis examined the potential for bias after accounting for the effects of nonresponse weight adjustments. The characteristics examined were gender, race/ethnicity, relative age, National School Lunch Program eligibility, student disability (SD) status, and English learner (EL) status.

Based on the student characteristics available, there does not appear to be evidence of substantial potential bias resulting from student nonresponse. Please see the full report for more details.

Printer-friendly Version