NAEP Technical DocumentationDefining Variance Strata and Forming Replicates for the 2011 Assessment

In the NAEP 2011 assessment, replicates were formed separately for each sample indicated by grade (4, 8, 12), school type (public, private), and assessment subject (mathematics, reading, science, computer-based writing). To reflect the school-level finite population corrections in the variance estimators for the two-stage samples used for mathematics, reading, and science assessments, variance strata were formed at both the school and student levels for these samples. For the computer-based writing assessments, which did not use school-level finite population corrections in the variance estimators, variance strata were formed only at the first-stage level.

The first step in forming replicates was to create variance strata in each primary stratum. In 2011, the mathematics, reading, and science assessments required formation of variance strata at both the school and student levels. The computer-based writing assessments required formation of variance strata only at the first-stage level.

The next step was to sort the appropriate sampling unit (school or student) in the order of its selection within the primary stratum and then pair off into preliminary variance strata. Sorting sample units by their order of sample selection reflects the implicit stratification and systematic sampling features of the sample design. Within each primary stratum with an even number of sampling units, all of the preliminary variance strata consisted of pairs of sampling units. However, within primary strata with an odd number of sampling units, all but one variance strata consisted of pairs of sampling units, while the last one consisted of three sampling units.

If there were more than 62 preliminary variance strata within a primary stratum, the preliminary variance strata were grouped to form 62 variance strata. This grouping effectively maximized the distance in the sort order between grouped preliminary variance strata. The first 62 preliminary variance strata, for example, were assigned to 62 different final variance strata in order (1 through 62), with the next 62 preliminary variance strata assigned to final variance strata 1 through 62, so that, for example, preliminary variance stratum 1, preliminary variance stratum 63, preliminary variance stratum 125 (if in fact there were that many), etc., were all assigned to the first final variance stratum.

If, on the other hand, there were fewer than 62 preliminary variance strata within a primary stratum, then the number of final variance strata was set equal to the number of preliminary variance strata. For example, consider a primary stratum with 111 sampled units sorted in their order of selection. The first two units were in the first preliminary variance stratum; the next two units were in the second preliminary variance stratum, and so on, resulting in 54 preliminary variance strata with two sample units each (doublets). The last three sample units were in the 55th preliminary variance stratum (triplet). Since there are no more than 62 preliminary variance strata, these were also the final variance strata.

Within each preliminary variance stratum containing a pair of sampling units, one sampling unit was randomly assigned as the first variance unit and the other as the second variance unit. Within each preliminary variance stratum containing three sampling units, the three first-stage units were randomly assigned variance units 1 through 3.

Reading, Mathematics, and Science Assessments

As described above, the mathematics, reading, and science assessments required variance strata at both the school and student level.

At the school-level for these samples, formation of variance strata did not pertain to certainty schools, since they are not subject to sampling variability, but only to noncertainty schools. The primary stratum for noncertainty schools was the highest school-level sampling stratum variable listed below, and the order of selection was defined by sort order on the school sampling frame.

• Trial Urban District Assessment (TUDA) districts, remainder of states (for states with TUDAs), or entire states for the public school samples at grades 4 and 8; and

• Private school affiliation (Catholic, Lutheran, Conservative Christian, and Other private) for the private school samples at grades 4 and 8.

At the student-level, all students were assigned to variance strata. The primary stratum was school, and the order of selection was defined by session number and position on the administration schedule.

Computer-Based Writing Assessments

As described above, variance strata for the computer-based writing assessments were formed at the first-stage sampling level and so differed by certainty and noncertainty PSUs. For noncertainty PSUs, the first-stage sampling units were PSUs, and the primary stratum was the combination of region and metropolitan status (MSA or non-MSA). For certainty PSUs, the first-stage sampling units were schools, and the primary stratum was school type (public or private).

For noncertainty PSUs, where only one PSU was selected per PSU stratum, variance strata were formed by pairing sampled PSUs with similar stratum characteristics within the same primary stratum (region by metropolitan status). This was accomplished by first sorting the 38 sampled PSUs by PSU stratum number and then grouping adjacent PSUs into 19 pairs. The values for a PSU stratum number reflect region and metropolitan status, as well as demographic and socioeconomic characteristics such as percentage of Black youth and percentage of children whose family income is below the poverty threshold. The formation of these 19 variance strata in this manner models a design of selecting two PSUs with probability proportional to size with replacement from each of 19 strata.

For certainty PSUs, the first stage of sampling is at the school level, and the formation of variance strata must reflect the sampling of schools within the certainty PSUs. Variance strata were formed by sorting the sampled schools in the 29 certainty PSUs by their order of selection within a primary stratum (school type) so that the sort order reflected the implicit stratification (region, locality type, minority status, and student enrollment for public schools; and region, private school type, and student enrollment size for private schools) and systematic sampling features of the sample design.

The first-stage units were then paired off into 43 preliminary variance strata. Within each primary stratum with an even number of first-stage units, all of the preliminary variance strata were pairs, and within primary strata with an odd number of first-stage units, one of the variance strata was a triplet (the last one), and all others were pairs.

If there were more than 43 preliminary variance strata within a primary stratum, the preliminary variance strata were grouped to form 43 variance strata. This grouping effectively maximized the distance in the sort order between grouped preliminary variance strata. The first 43 preliminary variance strata, for example, were assigned to 43 different final variance strata in order (1 through 43), with the next 43 preliminary variance strata assigned to final variance strata 1 through 43, so that, for example, preliminary variance stratum 1, preliminary variance stratum 44, preliminary variance stratum 87 (if there were that many), etc., were all assigned to the first final variance stratum. The final variance strata for the schools in the certainty PSUs were 1 through 43.

Within each pair of preliminary variance strata, one first-stage unit, designated at random, was assigned as the first variance unit and the other first-stage unit as the second variance unit. Within each triplet preliminary variance stratum, the three schools were randomly assigned variance units 1 through 3.

Printer-friendly Version