Table of Contents  |  Search Technical Documentation  |  References

HomeTDWNAEP Sample DesignSample Design for the 2001 AssessmentNational Main Assessment Sample Design in 2001Sampling of Schools in the 2001 National Main AssessmentSchool Substitution for Adjusting School Nonresponse for the 2001 National Main Assessment

NAEP Technical DocumentationSchool Substitution for Adjusting School Nonresponse for the 2001 National Main Assessment

			Substitute Frame Standard Deviations Originally Sampled Schools With and Without Substitutes

NAEP assessments traditionally use substitute schools to compensate for school-level nonresponse and to improve the overall yield of assessed students. Potential substitute schools were selected for all sampled schools in the 2001 national main assessment where a close match could be identified. The procedure for identifying substitutes followed the new school substitution procedure first used for the 2000 assessment. No sampled school was assigned more than one substitute, and no school was assigned to be a substitute for more than one school. The criteria for assigning substitutes were quite strict; many sampled schools were not assigned substitutes at all as there were no schools that met the necessary criteria to be a substitute.

All schools with either a fourth, eighth, or twelfth grade that were not in any of the samples were the potential pool of substitutes. The substitutes for the national main assessment schools and the mathematics online special study schools were drawn in a single step from a joint pool of unsampled schools. The twelfth -grade sampled schools received substitutes first, followed by eighth-grade sampled schools (for both the national main and the special study samples) next, and fourth-grade schools last (for both the national main and the special study samples). Twelfth-grade sampled schools were assigned substitutes first, as there are fewer schools available as substitutes at that grade.

It should be noted that a national main assessment school that was sampled for one grade was eligible to be a substitute for a national main assessment school in another grade, if it had that grade. A school with a particular sample type was not eligible to be a substitute for a school with a different sample type, as this would add expenses to the accommodations process. Mathematics online special study schools for one grade were also eligible to be substitutes for mathematics online special study schools in the other grade. National main assessment schools were not eligible to be substitutes for mathematics online special study schools, and vice versa.

There were a number of other "absolute" boundaries, as follows:

• Sampled schools could only have as substitutes schools in the same school type group, with school type groups defined as regular public, Bureau of Indian Affairs, Department of Defense, other public, Catholic, non-Catholic religious, and other private.

• Public schools could only have as substitutes schools in the same primary sampling unit (PSU) and with the same locality type.

• Catholic schools could only have as substitutes schools in the same district (usually diocese).

Within the various cells defined by these absolute boundaries, distance measures were computed for each sampled school and its potential matches. The distance measure was computed based on percent Black students, percent Hispanic students, and the square root of the grade enrollment. It used a standardized Euclidean distance D_ij as follows (with D_ij the distance between the i^th sampled school and the j^th candidate substitute):

with

• B_i = the percentage of Black students in school i,

• B_j = the percentage of Black students in out-of-district substitute school j,

• H_i = the percentage of Hispanic students,

• H_j = the percentage of Hispanic students in out-of-district substitute school j,

• E_i = the school grade enrollment,

• VBH = the average of the mean variance of Black percentages across all schools within the grade and the mean variance of Hispanic percentages across all schools within the grade, and

• VSQE = the mean variance of the square root of estimated grade enrollment across all schools within the grade.

Two passes were carried out for each grade. The first pass was a matching of regular public schools to schools outside their district. From a field operations viewpoint, it is preferable to have substitutes in different districts, as many public school districts refuse for all of their schools. An out-of-district substitute j was taken for a sampled public school i if the pair had a D_ij value less than .65. If there was more than one candidate for a particular sampled school i, then the j-school with the smallest D_ij was selected.

Although the selected cutoff point of .65 on the distance measure is somewhat arbitrary, statisticians in the 1994 Trial State Assessment (Mazzeo, Allen, and Kline 1995) selected this as the distance measure at which substitutes began to appear unacceptable. This cutoff point was also utilized in the 2000 assessment.

Note that a particular j-school might be the best substitute for two or more sampled schools. In this case, the school was matched with the sampled school in which it had the smallest D_ij value. Because of this rule, some sampled schools did not receive as substitutes their best matches, as those schools might have already been assigned to other sampled schools.

All matched schools in the first pass were removed from further consideration. The second pass was then a matching of regular public schools within their districts, and of all private schools (Catholic schools within their dioceses, religious non-Catholic schools in one cell, and non-religious private schools in one cell). The D_ij limit was 1.0; if a sampled school had no potential substitute lower than this limit, then that sampled school received no substitute. Note that a school might have a j-school with a D_ij value less than 1.0 that is used by another school as its substitute, and may thereby have no substitute.

Printer-friendly Version