NAEP Technical DocumentationEvaluation of State Achievement Data in the Sampling Frame for the 2009 State Assessment

The purpose of this analysis was to determine whether public schools selected for the 2009 samples were representative of the schools on the NAEP sampling frames in terms of student achievement. Percentiles of the achievement distributions were compared between the frame and sample schools for each public school jurisdiction in grades 4, 8, and 12.

Achievement Data

The achievement variable used in the analysis was the same variable used in the NAEP sample design to stratify the public school frame. For most jurisdictions, the variable was an achievement score provided by the jurisdiction. However, for some jurisdictions where achievement data were not available, median household income from the 2000 Census was used. (In 2000, the Census determined median household income based on the five-digit ZIP code area in which the school was located.) The achievement data consisted of various types of school-specific achievement measures from state assessment programs. The type of achievement data available varied by jurisdiction. For instance, in some states, the measure was the average score for a given state assessment. In other states, the measure was a percentile rank or percentage of students above a specific score. For grade 12, median household income from the 2000 Census was used.

During frame development, not every record on the Common Core of Data (CCD) file matched to the achievement data files created for the National Center for Education Statistics (NCES), even in jurisdictions where those data were generally available. For schools that did not match, their achievement score was imputed by a mean matching imputation approach using the mean achievement score for schools with complete achievement data within the same jurisdiction/urbanicity/race/ethnicity stratum combination.

Methodology

To determine whether the distributions between the frame and sample schools were different, comparisons of percentile estimates were made for the 10th, 25th, 50th, 75th, and 90th percentile levels as well as the mean for each public school jurisdiction by grade. Frame and sample school estimates were considered statistically different if the frame value fell outside the 95 percent confidence interval of the corresponding sample estimate. The percentile values for the frame schools were calculated by weighting each school by the estimated number of students in the given grade. The percentile estimates for the sample schools were calculated using school weights and weighted by the school measure of size (estimated number of students in the given grade). The 95 percent confidence intervals for the school sample estimates were calculated in WesVar—software for computing estimates of sampling variance from complex sample survey (Westat, 2000b)—using the Woodruff method (Sarndal, Swensson, and Wretman 1992) and without the use of a finite population correction factor. A finite population correction is not traditionally used in computing variances for NAEP estimates.

Results

As mentioned above, sample and frame achievement distributions were determined to be different if at least one of the percentile estimates or the mean differed significantly at the 95 percent confidence level. Out of all the jurisdiction and grade comparisons, only 11 of the 738 distributions compared were found to be significantly different. They are shown in the table below.

Summary of significant differences in achievement measures between the sample and the frame, by grade and jurisdiction, state assessment: 2009
Grade	Jurisdiction	Achievement data / median income	Estimate	Frame	Sample	Confidence interval
SOURCE: U.S. Department of Education, Institute of Education Sciences, National Center for Education Statistics, National Assessment of Educational Progress (NAEP), 2009 Assessment.
4	Alabama	Achievement data	mean	43.81	42.22	(40.78; 43.66)
4	Oklahoma	Achievement data	25th percentile	746.78	744.16	(738.39; 745.71)
8	Alaska	Median income	10th percentile	36,361.00	36,022.00	(35,890; 36,199)
	Michigan	Achievement data	10th percentile	33.82	34.90	(34.23; 35.69)
	Michigan	Achievement data	25th percentile	52.02	53.90	(52.27; 55.79)
	New Mexico	Achievement data	90th percentile	71.55	69.95	(69.4; 70.51)
	Pennsylvania	Achievement data	10th percentile	40.18	37.61	(36.35; 38.74)
	Washington	Achievement data	50th percentile	50.27	51.75	(50.31; 52.42)
	Detroit TUDA	Achievement data	90th percentile	60.77	60.18	(59.45; 60.71)
12	New Hampshire	Median income	90th percentile	68,772.46	68,719.19	(68,690.15; 68,748.23)
12	West Virginia	Median income	50th percentile	28,868.35	29,060.92	(28,869.2; 29,354.21)

The number of significant differences found in this analysis is smaller than what would be expected to occur by chance, given the large number of comparisons that were made. The small number of significant differences may be partially accounted for by the lack of use of a finite population correction factor in the calculation of the sampling variances. However, the close adherence of sample values to frame values suggests that there is little evidence that the school sample for NAEP 2009 is not representative of the frame from which it was selected. The achievement/median income variable is used as the third-level sort order variable in the school systematic selection procedure. While this variable was low in the sorting hierarchy, it still helps control how representative the sampled schools are in terms of achievement. The close agreement between frame and sample values of these achievement/median income variables provides assurance that the selected sample is representative of the frame with respect to achievement status.

Last updated 25 February 2016 (GF)

Printer-friendly Version