Skip to main content
IES    
NCES
National Center for
Education Statistics
Menu
Contact
Tools
ED.gov
Newsflash
Search IES

Table of Contents  |  Search Technical Documentation  |  References

NAEP Assessment Weighting Procedures → Weighting Procedures for the 2005 Assessment → Computation of Replicate Weights for the 2005 Assessment → Replicate Variance Estimation for the 2005 Assessment

NAEP Technical DocumentationReplicate Variance Estimation for the 2005 Assessment

Variances for NAEP assessment estimates are computed using the jackknife replicate variance procedure. This technique is applicable for common statistics, such as means and ratios, as well as for more complex statistics such as Item Response Theory (IRT) scores. 

In general, the jackknife replicate variance procedure involves pairing clusters of first-stage sampling units to form H variance strata (h = 1, 2, 3, ...,H) with two units per stratum. The first replicate is formed by deleting one unit at random from the first variance stratum, inflating the weight of the remaining unit to weight up to the variance stratum total, and using all other units from the other (H-1) strata. This procedure is carried out for each variance stratum resulting in H replicates, each of which provides an estimate of the population total.

The jackknife estimate of the variance for any given statistic is given by the following formula:

v left brace t with hat right brace equals summation with superscript H subscript h equals 1 left brace t hat subscript h minus t hat right brace square

where

  •  represents the full sample estimate of the given statistic, and
  • t hat subscript h represents the corresponding estimate for replicate h.

Each replicate undergoes the same weighting procedure as the full sample so that the jackknife variance estimator reflects the contributions to or reductions in variance resulting from the various weighting adjustments. 

The NAEP jackknife variance estimator is based on 62 variance strata resulting in a set of 62 replicate weights assigned to each school and student.

The basic idea of the jackknife variance estimator is to create the replicate weights so that use of the jackknife procedure results in a correct, unbiased variance estimator for simple totals and means which is also reasonably efficient (i.e., has a low variance as a variance estimator). The jackknife variance estimator will then produce a consistent (but not fully unbiased) estimate of variance for (sufficiently smooth) nonlinear functions of total and mean estimates such as ratios, regression coefficients, and so forth ( Shao and Tu, 1995). The development below shows why the NAEP jackknife variance estimator returns a correct unbiased variance estimator for totals and means, which is the cornerstone to the asymptotic results for nonlinear estimators. See for example Rust (1985). This paper also discusses why this variance estimator is generally efficient (i.e., more reliable than alternative approaches requiring similar computational resources).

The development will be done for an estimate of a mean based on a simplified sample design which closely approximates the sample design for first-stage units used in the NAEP studies. The sample design is a stratified random sample with H strata with population weights Wh, stratum sample sizes nh, and stratum sample means y bar subscript h . The population estimator upper case y hat bar and standard unbiased variance estimator v left brace upper case y hat bar right brace are:

Cap y hat bar equals summation from h equals one to cap open paren h cap w subscript h times y bar subscript h close paren

with

s subscript h squared equals open bracket one divided by open paren n subscript h minus one close paren close bracket times summation from i equals one to n subscript h open paren y subscript h subsubscript i minus y bar subscript h close paren squared

The jackknife replicate variance estimator assigns one replicate h=1,…,H  to each stratum, so that the number of replicates equals H. In NAEP, the replicates correspond generally to doublets and triplets (with the latter only being used if there are an odd number of sample units within a particular hard boundary generating replicate strata). For doublets, the process of generating replicates can be viewed as taking a simple random sample of size nh/2 within the replicate stratum, and assigning a doubled weight to the sampled elements, and zero weight to the unsampled elements. In this simplified context of stratified random sampling, this assignment reduces to replacing y bar subscript h with y bar subscript h times open paren cap j close paren, the latter being the sample mean of the sampled nh/2 units. The replicate estimator corresponding to stratum r is

cap y hat bar open paren r close paren equals summation from h does not equal r to cap h cap w subscript h times y bar subscript h plus cap w subscript r times y bar subscript r open paren cap j close paren

The r-th term in the sum of squares for v subscript j open paren cap y hat bar close paren is thus:

open paren cap y hat bar open paren r subscript one close paren minus cap y hat bar close paren squared equals cap w subscript r squared times open paren y bar subscript r subsubscript one open paren cap j close paren minus y bar subscript r close paren square

In stratified random sampling, when a sample of size nr/2 is drawn without replacement from a population of size nr, the sampling variance is

cap e open paren y bar subscript r open paren cap j close paren minus y bar subscript r close paren squared equals open bracket one divided by open paren n subscript r divided by two close paren close bracket times open bracket open subbracket n subscript r minus open paren n subscript r divided by two close paren close subbracket divided by n subscript r close bracket times open bracket one divided by open paren n subscript r minus one close paren close bracket times summation from i equals one to n subscript r open paren y subscript r subsubscript i minus y bar subscript r close paren squared equals open bracket one divided by open subbracket n subscript r times open paren n subscript r minus one close paren close subbracket close bracket times summation from i equals one to n subscript r open paren y subscript r subsubscript i minus y bar subscript r close paren squared equals s subscript r squared divided by n subscript r

See for example Cochran (1977), Theorem 5.3, using nr as the “population size”, nr/2 as the “sample size”, and sr2 as the “population variance” in the given formula. Thus

cap e open bracket cap w subscript r squared times open paren y bar subscript r open paren cap j close paren minus y bar subscript r close paren squared close bracket equals cap w subscript r squared open paren s subscript r squared divided by n subscript r close paren

Taking the expectation over all of these stratified samples of size nr/2, it is found that

cap e open paren v subscript j open paren cap y hat bar close paren close paren equals v open paren cap y hat bar close paren

In this sense, the jackknife variance estimator “gives back” the sample variance estimator for means and totals as desired under the theory. In practice, random selection is not done in each replicate stratum, but units are instead assigned systematically (the first, third, etc.). Replicate strata are also grouped to make sure that the number of replicates is not too large (the replicate total is usually 62 for NAEP surveys). The randomization from the original sample distribution guarantees that the sum of squares contributed by each replicate will be close to the target expected value (rather than much larger or much smaller).

For triplets, the NAEP weighting contractor assigns two sets of replicate weights for replicate stratum r: r1 and r2 (which are then usually grouped with other doublets and/or triplets). Note that r1 is always equal to r, with r2 being another replicate (“far away” from the first replicate). The replicate stratum r1 is partitioned into three equal-sized replicate units, with the following replicate weight assignments for the two replicates:

w subscript i open paren r subscript one close paren equals one point five times w subscript i if i is an element of replicate stratum r and replicate unit one or equals one point five times w subscript i if i is an element of replicate stratum r and replicate unit two or equals zero if i is an element of replicate stratum r and replicate unit three or equals w subscript i if i is not an element of replicate stratum r

where wi is the full sample base weight,

w subscript i open paren r subscript two close paren equals one point five times w subscript i if i is an element of replicate stratum r and replicate unit one or equals zero if i is an element of replicate stratum r and replicate unit two or equals one point five times w subscript i if i is an element of replicate stratum r and replicate unit three or equals w subscript i if i is not an element of replicate stratum r

In the case of stratified random sampling, this formula reduces to replacing y bar r with y bar subscript r subsubscript one open paren cap j close paren  for replicate r1, where y bar subscript r subsubscript one open paren cap j close paren is the sample mean from a “2/3” sample of 2nr/3 units from the nr sample units in the replicate stratum, and replacing y bar rwith y bar subscript r subsubscript two open paren cap j close parenfor replicate r2, where  y bar subscript r subsubscript two open paren cap j close parenis the sample mean from another overlapping “2/3” sample of 2nr/3 units from the nr sample units in the replicate stratum.

The r1-th and r2-th replicates can be written:

cap y hat bar open paren r subscript one close paren equals summation from h does not equal r to cap h cap w subscript h times y bar subscript h plus cap w subscript r times y bar subscript r subsubscript one open paren cap j close paren

cap y hat bar open paren r subscript two close paren equals summation from h does not equal r to cap h cap w subscript h times y bar subscript h plus cap w subscript r times y bar subscript r subsubscript two open paren cap j close paren

From these formulas, expressions for the r1-th and r2-th components of the jackknife variance estimator  are obtained (ignoring other sums of squares from other grouped components attached to those replicates):

open paren cap y hat bar open paren r subscript one close paren minus cap y hat bar close paren squared equals cap w subscript r squared times open paren y bar subscript r subsubscript one open paren cap j close paren minus y bar subscript r close paren squared

open paren cap y hat bar open paren r subscript two close paren minus cap y hat bar close paren squared equals cap w subscript r squared times open paren y bar subscript r subsubscript two open paren cap j close paren minus y bar subscript r close paren squared

These sums of squares have expectations as follows, using the general formula for sampling variances:

cap e open paren y bar subscript r subsubscript one open paren cap j close paren minus y bar subscript r close paren squared equals open bracket one divided by open paren two n subscript r divided by three close paren close bracket open bracket times open subbracket n subscript r minus open paren two n subscript r divided by three close paren close subbracket divided by n subscript r close bracket times open bracket one divided by open paren n subscript r minus one close paren close bracket times summation from i equals one to n subscript r open paren y subscript r subsubscript i minus y bar subscript r close paren squared equals open bracket one divided by open subbracket two n subscript r times open paren n subscript r minus one close paren close subbracket close bracket times summation from i equals one to n subscript r open paren y subscript r subsubscript i minus y bar subscript r close paren squared equals s subscript r squared divided by two times n subscript r

cap e open paren y bar subscript r subsubscript two open paren cap j close paren minus y bar subscript r close paren squared equals open bracket one divided by open paren two n subscript r divided by three close paren close bracket times open bracket open subbracket n subscript r minus open paren two times n subscript r divided by three close paren close subbracket divided by n subscript r close bracket times open bracket one divided by open paren n subscript r minus one close paren close bracket times summation from i equals one to n subscript r open paren y subscript r subsubscript i minus y bar subscript r close paren squared equals open bracket one divided by open subbracket two times n subscript r times open paren n subscript r minus one close paren close subbracket close bracket times summation from i equals one to n subscript r open paren y subscript r subsubscript i minus y bar subscript r close paren squared equals s subscript r squared divided by two times n subscript r

Thus,

cap e open bracket cap w subscript r squared times open paren y bar subscript r subsubscript one open paren cap j close paren minus y bar subscript r close paren squared plus cap w subscript r squared open paren y bar subscript r subsubscript two open paren cap j close paren minus y bar subscript r close paren squared close bracket equals cap w subscript r squared times open bracket open paren s subscript r squared divided by two n subscript r close paren plus open paren s subscript r squared divided by two n subscript r close paren close bracket equals cap w subscript r squared times open paren s subscript r squared divided by n subscript r close paren

as desired again.

Last updated 27 October 2009 (JL)

Printer-friendly Version


IES    
NCES
National Center for
Education Statistics
Explore the Institute of Education Sciences