NAEP Technical DocumentationSelecting Substitutes for the 2003 State Assessment

The pool of potential substitutes for each originally sampled school¹ had to satisfy the following criteria:

The school could not be in the original school sample.
The school had to be in the same state and same urbanicity/Trial Urban District Assessment (TUDA) stratum as the original school and be the same school type.
At grade 4 schools were grouped by whether they were charter schools.
The school could not be classified as a closed or ineligible school, as determined by the
new-school district canvassing process.

After identifying the pool of potential substitutes for each sampled school, a distance measure was computed for each original school-potential substitute pair. Using indexes h as the state-urbanicity/TUDA stratum-district type cell, i as the sampled school, and j as the potential substitute school, the distance measure DM_hij was computed as follows:

with

MIN1 representing the percentage in the most important minority in the state-urbanization/TUDA cell,
MIN2 representing the percentage in the second most important minority in the state-urbanization/TUDA cell,
EST representing the estimated grade enrollment,
AM representing the achievement score or ZIP Code area median income,
VAR_M representing the average of the variance of MIN1 over all schools and the variance of MIN2 over all schools,
VAR_SQE representing the variance of the square root of grade enrollment over all schools, and
VAR_AM representing the variance of the achievement score/median income over all schools.

On the first pass, a check was made for each school for a substitute which was out-of-district² and had an upper bound for DM_hij of 0.6. Note that this upper bound implies at the very least that the substitute would be at a distance less than 60 percent of this average standard deviation (averaged over the four characteristics). Each potential substitute could replace only one school, so substitutes were chosen one at a time, based on the best original-substitute pair available at that point.

The selection of a set of substitutes can be illustrated via the example given below, which uses five sampled schools and 10 potential substitutes. The entries in the table are distance measures in percentage points. School i=5 is a good substitute for both j=2 and j=4, but gets selected for j=4 because its distance measure is smaller for j=4. The order of selected pairs is (j=4, i=5), (j=3, i=4), (j=5, i=1), (j=1, i=6), and (j=2, i=9). The selected pairs are indicated in bold in the table.

Substitute selection example, state assessment, by potential substitutes: 2003
Potential substitutes	Sampled schools
Potential substitutes	j=1	j=2	j=3	j=4	j=5
i=1	26	57	40	30	16
i=2	42	47	31	55	48
i=3	36	32	51	46	54
i=4	41	27	12	59	34
i=5	50	14	39	10	69
i=6	22	43	28	61	44
i=7	56	63	58	64	45
i=8	68	49	33	35	65
i=9	52	25	62	67	60
i=10	29	66	38	53	37
SOURCE: U.S. Department of Education, Institute of Education Sciences, National Center for Education Statistics, National Assessment of Educational Progress (NAEP), 2003.

At the end of this process, it usually is the case that not every original school will find a substitute (even if potentials might have been available, as they could have been chosen for other original schools). Accordingly, a second pass was carried out, in which in-district as well as out-of-district substitutes were allowed, and an upper bound for DM_hij of 0.75 was set. Many original schools found no substitute even after this second pass, and were left without one.

¹ Originally sampled new schools did not receive substitutes.
² It was preferred that substitutes be chosen from a different district than the original school, as noncooperation tended to cluster in particular districts (due sometimes to decisions made at the district level).

Last updated 14 January 2009 (RF)

Printer-friendly Version