NAEP Technical DocumentationNonresponse Bias Analyses for the 2008 Arts Assessment

The analyses suggest that although school nonresponse was quite high for private schools, overall there does not appear to be evidence of substantial bias. After accounting for school substitution and school-level nonresponse weight adjustments, the results for race/ethnicity and private school type remain significant. Specifically, the percentage for Hispanics remains significant after substitution, with a small absolute bias of about 2 percent and a moderate relative bias of about 22 percent. The percentage White, non-Hispanic becomes significant only after the nonresponse adjustment, with a small absolute bias of 4.3 percent on an estimate of about 81 percent.  This level of bias is reasonable, since higher priority was given to preserving affiliation type over race/ethnicity during nonresponse adjustments, due to the small number of private schools in the arts sample.

The result for private school type remains significant, after accounting for the effects of substitution and nonresponse adjustment.  In comparison to the original sample estimate of 61 percent in the Catholic schools, about 73 percent of the original sample respondents are Catholic schools. After substitution, the Catholic school estimate decreases to about 71 percent, which reduces the absolute bias from 12 percent to about 10 percent. The absolute bias is further reduced to about 7 percent after the nonresponse adjustment, with a Catholic school estimate of about 67 percent.  Due to the small sample size, affiliation was not able to be preserved in the Northeast and the West during the nonresponse adjustment. Therefore, while the discrepancy in response rate between the affiliation types is addressed to some degree, it is not entirely accounted for by substitution and nonresponse adjustment.