Table of Contents  |  Search Technical Documentation  |  References

NAEP Sample Design → NAEP 2002 Sample Design → 2002 State Assessment Sample Design → Substitute Schools → Selection of Substitute Schools

NAEP Technical DocumentationSelection of Substitute Schools

The pool of potential substitutes for each originally sampled school¹ had to satisfy the following criteria:

• The school could not be in the alpha sample.

• The school could not be in the beta sample in the same grade.

• The school had to be in the same state and same urbanicity stratum as the original school.

• The school had to be in the same district as the original school, if the district was one which contained over 20 percent of the state’s enrollment.

• The school could not be indicated in the new school district canvassing process as a closed or ineligible school.

After identifying the correct pool of potential substitutes for each sampled school, a distance measure was computed for each original school-potential substitute pair. Using as indexes h as the state-urbanicity stratum-district type cell, i as the sampled school, and j as the potential substitute school, the distance measure DM_hij was computed as follows:

with

• MIN1 the percentage of the largest minority in the state-urbanization cell,

• MIN2 the percentage in the second largest minority in the state-urbanization cell,

• EST the estimated grade enrollment,

• AM the state test achievement score or county median income (state test achievement score if available, median income otherwise),

• VAR_M the average of the variance of MIN1 over all schools and the variance of MIN2 over all schools,

• VAR_SQE the variance of the square root of grade enrollment over all schools, and

• VAR_AM the variance of the state test achievement score/median income over all schools.

This distance measure was a measure of how much a pair of schools is 'alike': for example, a distance measure of zero would indicate that two schools have the same minority percentage (largest and second largest minority), the same enrollment, and the same achievement score or median income. A large value of the distance measure would indicate a pair of schools with minority percentages, enrollments, and/or achievement scores or median incomes at opposite extremes: 'very different' schools. If for example a pair of schools has largest and second largest minority percentages, square root estimated enrollments, and achievement scores which are all exactly one standard deviation apart, then the distance measure was equal to the square root of 4, i.e., 2. We used a square root transformation for the estimated enrollment as these tend to differ by orders of magnitude (we could have also used a logarithm transformation, but this would be too strong).

On the "first pass" each school was checked for a substitute which was out-of-district,² and had an upper bound for DM_hij of 0.6. Note that this upper bound implies at the very least that the substitute was at a distance less than 60 percent of the overall standard deviation for each of the four characteristics. Each substitute could be a substitute for only one school, so substitutes are selected one at a time, based on the "best" original-substitute pair available at that point.

The assignment process can be illustrated via the example given below, in which we have 5 sampled schools and 10 potential substitutes. We check the distances for all possible pairs, which are indicated below by subscripts i and j (i=1,...,10 for the potential substitutes and j=1,...5 for the sampled schools). The numbers in the table are the distances, with the bold-faced value the selected substitute in the column (the best potential substitute for that sampled school j).School i=5 is a good substitute for both j=2 and j=4, but gets selected for j=4. The selected pairs in this example (in their order of selection) are (j=4, i=5), (j=3, i=4), (j=5, i=1), (j=1, i=6), and (j=2, i= 9).

Potential substitutes	Sampled schools
	j=1	j=2	j=3	j=4	j=5
i=1	26	57	40	30	16
i=2	42	47	31	55	48
i=3	36	32	51	46	54
i=4	41	27	12	59	34
i=5	50	14	39	10	69
i=6	22	43	28	61	44
i=7	56	63	58	64	45
i=8	68	49	33	35	65
i=9	52	25	62	67	60
i=10	29	66	38	53	37

At the end of this process, it usually is the case that not every original school will find a substitute (even if potentials might have been available, as they could have been chosen for other original schools). A "second pass" that allowed for in-district substitutes as well as out-of-district substitutes was carried through, and an upper bound for DM_hij of 0.75 was set. Many original schools had no substitute even after this second pass, and were left as such. The procedure then is conservative in that it doesn't provide substitutes for every sampled school, but the substitutes that are provided are guaranteed to be 'similar' in terms of ethnicity, enrollment, and achievement score or median income.

¹ Originally sampled new schools did not receive substitutes.
² Substitutes that are out-of-district are preferred, as noncooperation tends to cluster in particular districts (due sometimes to decisions taken at the district level).

Printer-friendly Version