HomeTDWNAEP Assessment Weighting ProceduresWeighting Procedures for the 2009 AssessmentQuality Control on Weighting Procedures for the 2009 AssessmentNonresponse Bias Analyses for the 2009 Assessment

NAEP Technical DocumentationNonresponse Bias Analyses for the 2009 Assessment

School-level nonresponse bias analyses were conducted on private schools at all three grades (4, 8, and 12),  Bureau of Indian Education (BIE) schools at grade 8, and Illinois public schools at grade 12 since the school-level participation rate of each of these reporting groups fell below the NCES statistical standard threshold of 85 percent participation. The school analysis was conducted in three parts. The first part of the analysis looked for the potential for nonresponse bias that was introduced through school nonresponse. The second part of the analysis examined the remaining potential for nonresponse bias after accounting for the mitigating effects of substitution. The third part examined the remaining potential for bias after accounting for the mitigating effects of both substitution and nonresponse weight adjustments.

Overall, it was found that the school nonresponse adjustments reduced the potential for nonresponse bias among several variables. For grades 4, 8, and 12 private schools, substitution and nonresponse adjustments appear to have been at least partially effective in reducing the nonresponse bias. At the overall reporting group level for each of the private school samples the biases were completely removed; however, the biases for some variables within each of the samples were still significant. For grade 4 private schools, the results for all characteristics were not significant after nonresponse adjustments, with the exception of race/ethnicity. The result for race/ethnicity variables was improved as the largest absolute bias and largest relative bias among all race/ethnicity variables were reduced, but the result for “percentage of Black, not Hispanic” changed from not significant to significant. For grade 8 private schools, after nonresponse adjustments, the results for school size and race/ethnicity were significant. For school size, the largest absolute bias stayed the same while the largest absolute bias among all race/ethnicity variables went from not significant to significant. For grade 12 private schools, while “private school reporting group” became not significant, the result for census region and “size of school attended by average student” became significant.

For grade 12 public schools in Illinois, substitution and nonresponse adjustments appear to have been effective in reducing the nonresponse bias, as all significant results changed to not-significant after substitution and nonresponse adjustments.

Student-level nonresponse bias analyses were conducted on the grade 8 public school student samples in the Detroit and Washington, D.C. TUDAs; on the overall grade 12 public school sample; and 9 out of the 11 grade 12 public school state samples (Connecticut, Florida, Illinois, Iowa, Massachusetts, New Hampshire, New Jersey, South Dakota, and West Virginia), since the student-level participation rates of these reporting groups fell below the NCES statistical standard threshold of 85 percent participation. The first part of the analysis indicated the potential for nonresponse bias that was introduced through student nonresponse. The second part of the analysis examined the potential for bias after accounting for the effects of nonresponse weight adjustments.

In general, the number of variables with significant differences decreased or stayed the same after nonresponse adjustments, with the exception of grade 12 public school students in Florida. Even when the test result was significant, the absolute value of the bias was reduced in most cases. After nonresponse adjustments, for all of the significant variables, the percentage difference was actually very small, 1.3 percent or less.  For example, for the subdomain with the lowest response rate (67.72 percent), grade 12 public schools in New Hampshire, free lunch eligibility was significant before and after nonresponse adjustments, but the bias was reduced from 1.0 to 0.7. The relative bias reduced from 8.1 percent to 5.5 percent. The significant result for relative age and SD status changed to not-significant after nonresponse adjustments. In summary, based on the student characteristics available, there does not appear to be evidence of substantial bias resulting from student nonresponse.

To see the entire report or detailed tables, see the NAEP 2009 nonresponse bias analysis report (755KB PDF).

Printer-friendly Version