Measuring Status and Change in NAEP Inclusion Rates of Students with Disabilities: Trial Urban District Assessment 2005 to 2007

• Trial Urban District Assessment
• Factors Affecting Inclusion
• Methods
• Results

In 2005, the U.S. Government Accountability Office (GAO) released the report No Child Left Behind Act: Most Students With Disabilities Participated in Statewide Assessments, but Inclusion Options Could Be Improved. The report focused primarily on No Child Left Behind and state tests. However, there was one chapter about the National Assessment of Educational Progress (NAEP). In the report, the GAO recommended that NAEP "work with the states, particularly those with high exclusion rates, to explore strategies to reduce the number of students with disabilities who are excluded from the NAEP assessment." The National Center for Education Statistics (NCES) responded with the following actions:

• researched the local decision-making process for participation and accommodation decisions of students with disabilities on NAEP;
• implemented a process to determine whether students could participate in NAEP without their normal state accommodations; and
• improved training of NAEP administrators and field staff for 2007 assessments that clarified the criteria for inclusion.

NCES also conducted research to develop a methodology for measuring state inclusion rates while taking into account the differing demographics and policies in each state. The report Measuring Status and Change in NAEP Inclusion Rates of Students With Disabilities was the result of that research. It provided a methodology and two measures of change in each state?s inclusion rate taking into consideration the following factors that differ across states and across time:

• the prevalence of students with different types and severities of disability; and
• the accommodations that states permit in their own testing programs compared with those allowed for NAEP.

That study reported results for 50 states and the District of Columbia and used data from the 2005 and 2007 NAEP fourth- and eighth-grade reading and mathematics assessments. The full research and development report, Measuring Status and Change in NAEP Inclusion Rates of Students With Disabilities, is available for download.

The methodology developed in the report Measuring Status and Change in NAEP Inclusion Rates of Students With Disabilities was next applied to measuring change in districts participating in the Trial Urban District Assessment (TUDA). As with this report, the discussions presented here are exploratory in nature and do not reach definitive conclusions at this time because the methodology is new and developing.

tudaTrial Urban District Assessment

The TUDA program is designed to explore the feasibility of using NAEP to report on the performance of public school students at the district level. As authorized by federal law, NAEP has administered the mathematics, reading, science, and writing assessments to samples of students in selected urban-district public schools.

Assessment results from the NAEP TUDA make it possible to compare the performance of students in participating urban school districts to that of public school students in the nation, in large central cities (population over 250,000), and to each other. In 2007, about 38,000 fourth- and eighth-graders from 11 urban districts participated in the mathematics assessment. An approximately equal number participated in the reading assessment. Participating districts were: Atlanta, Austin, Boston, Charlotte, Chicago, Cleveland, District of Columbia, Houston, Los Angeles, New York City, and San Diego. The District of Columbia participated in NAEP as both a state and as part of the TUDA program.

factorsFactors Affecting Inclusion

Inclusion rates are expected to vary by district depending on differences in a district?s population of students with disabilities. Whether a student can participate in NAEP depends on the student?s characteristics.

Student characteristics that are expected to have an impact on a district?s inclusion rate include the following:

• Type of disability;
• Severity of the disability;
• Grade level of instruction; and
• Receipt of an accommodation on state tests that would not be allowed on NAEP.

Students with less severe disabilities, such as a speech or hearing impairment, are more likely to be included in NAEP testing. Students with more severe disabilities, such as mental retardation, are less likely to be included in NAEP.

Controlling for differences in student characteristics

A district with a 90 percent inclusion rate is not necessarily more inclusive than a district with an 80 percent inclusion rate. If a district has a higher percentage of severely disabled students, for example, it would be expected to have a lower inclusion rate. Hence, to properly compare the status of inclusion rates across districts or to properly measure a district?s change in inclusion rates across time, differences and changes in districts? populations of students with disabilities must be controlled. For example, if a district experienced a drop in the proportion of students classified as having mental retardation, who are less likely to be included, the district?s inclusion rate would be expected to increase.

Variation in the population of students with disabilities across districts

The characteristics of a student with disabilities affect the likelihood that the student will be included in NAEP. The percentage of students with each of these characteristics varies across districts. In the tables below, the percent of students with each characteristic are present for participating urban districts for a sample subject, grade, and year: mathematics, grade 4, 2007. All percentages discussed are based on students identified as having a disability who are not English learners.

The first table gives the number of observations used for analysis in each district. This is the un-weighted number of students observed in each district that have a disability but are not an English learner. The percentages that follow in this and the next two tables are the weighted percentage of those students with disabilities with each characteristic. In this first table are percentages of students with each disability type. These types are not mutually exclusive. Additionally, while the NAEP questionnaire allows respondents to identify 11 different disability types as well as an ?other? category, many of the types were uncommon. For our statistical analysis we collapsed all but the four largest disability types into the ?other? category.

• Disability types
• Severity of disability
• Grade level of instruction and receipt of an accommodation on the state test that would not be allowed on NAEP

methodsMethods

Measuring change

To measure change in district-level inclusion rates, researchers developed two approaches for controlling for differences in populations of students with disabilities. The nation-based approach uses national averages to set expectations for including different types of students. The jurisdiction-specific approach uses averages in each jurisdiction to set expectations for including different types of students for that jurisdiction. Each approach has its advantages and disadvantages. We report results for both without claim that either is better (see the state-level report, Measuring Status and Change in NAEP Inclusion Rates of Students With Disabilities, for a full explanation of the methodology).

Nation-based approach

The nation-based approach uses average inclusiveness by type of student among all jurisdictions in the nation as a benchmark for comparing status and measuring change. The national inclusion rate was examined for each of the student characteristics in 2005 to determine the proportion of each that was included in NAEP. These rates were used to predict rates of inclusion for each student in the district. For example, if nationally 90 percent of students identified as having a speech impairment were included in NAEP, it was expected that in each district in each year, 2005 and 2007, 90 percent of students identified as having a speech impairment would be included. Change is measured relative to each year?s prediction. The benefit of the nation-based approach is that because of the large number of observations we are able to cross the disability types with the severity levels and with the grade level of instruction in our statistical model that estimates the benchmark averages. This allows us to estimate an independent benchmark for each combination of those characteristics.

Jurisdiction-specific approach

For state-level analysis, we developed an approach titled the state-specific approach. This approach can be applied equally to any identified jurisdiction. Here, within the context of applying the methods of the state-level analysis to districts participating in the TUDA, we use a more generic name for this approach: the jurisdiction-specific approach. In the jurisdiction-specific approach, a district?s average 2005 inclusion rates for different student types were used as benchmarks for measuring change. These averages were used to predict the district?s inclusion rate in 2007. For example, if in District A 90 percent of students identified as having a speech impairment were included in NAEP in 2005, it was expected that in District A in 2007, 90 percent of students identified as having a speech impairment would be included. In District B, however, the benchmark set in 2005 might be 85 percent, and this would be the expectation for District B for 2007. The predicted inclusion rate for 2007 was then compared to the actual rate in 2007. If the actual inclusion rate was greater than the predicted rate, the district was considered to have made progress. In the jurisdiction-specific approach, we are able to cross the disability types with the severity levels in our statistical model that estimates the benchmark averages. While there is less independence among the benchmarks for different types of students, the benefit is that a separate set of benchmarks are estimated for each district.

Status of inclusion rates—a context for understanding change

Each district?s change measure needs to be understood in terms of how inclusive the district was in the initial year, 2005. Districts with high inclusion rates in the initial year have relatively less room for improvement, while districts with lower inclusion rates at the start have more room for improvement. A comparison of the status of inclusion rates in 2005, the starting point, is essential for understanding the change measure.

To compare inclusion rates at the starting point, we need to control for differences across districts in the populations of students with disabilities. To do so, we use the averages from the nation-based approach and compare inclusiveness across districts relative to each district?s expected inclusion rate: a district with an actual inclusion rate in 2005 greater than predicted is considered more inclusive while a district with an actual inclusion rate less than predicted is considered less inclusive.

resultsResults

The following tables contain results for the nation-based approach. Standard errors were calculated using NAEP jackknife weights as described in the state-level report. Tests for statistical significance were conducted at the .05 level using the Student-t distribution to ensure that the differences are larger than those that might be expected due to chance or sampling error.

Reading Grade 4

• Districts with significant change
  • Few districts—3 out of 11—had significant change.

• Significantly more inclusive
  • No district was significantly more inclusive.

• Significantly less inclusive
  • Three districts under the nation-based approach were significantly less inclusive.

Reading Grade 8

• Districts with significant change
  • Over half of the districts—6 out of 11—had significant change.

• Significantly more inclusive
  • Only one district was significantly more inclusive.

• Significantly less inclusive
  • Five districts under the nation-based approach were significantly less inclusive.

Mathematics Grade 4

• Districts with significant change
  • Few districts—2 out of 11—had significant change.

• Significantly more inclusive
  • One district was significantly more inclusive.

• Significantly less inclusive
  • One district was significantly less inclusive.

Mathematics Grade 8

• Districts with significant change
  • Most districts—7 out of 11—had significant change.

• Significantly more inclusive
  • Only one district was significantly more inclusive.

• Significantly less inclusive
  • Six districts were significantly less inclusive.

The following tables contain results for the jurisdiction-specific approach (note: in the jurisdiction-specific approach, the 2005 predicted is always equal to the 2005 actual by design; we therefore do not report the 2005 predicted value or the 2005 difference from predicted, which is always zero):

Reading Grade 4

• Districts with significant change
  • Few districts—3 out of 11—had significant change.

• Significantly more inclusive
  • Only one district was significantly more inclusive and was so only under the jurisdiction-specific approach.

• Significantly less inclusive
  • Of the three districts that were significantly less inclusive under the nation-based approach, two were significantly less inclusive under the jurisdiction-specific approach.

Reading Grade 8

• Districts with significant change
  • Few districts—2 out of 11—had significant change.

• Significantly more inclusive
  • While one district was significantly more inclusive under the nation-based approach, none were so under the jurisdiction-specific approach.

• Significantly less inclusive
  • Of the five districts that were significantly less inclusive under the nation-based approach, only two were under the jurisdiction-specific approach.

Mathematics Grade 4

• Districts with significant change
  • Few districts—2 out of 11—had significant change.

• Significantly more inclusive
  • One district was significantly more inclusive only under the jurisdiction-specific approach. This is not the same district that was significant under the nation-based approach.

• Significantly less inclusive
  • One district was significantly less inclusive, and was so under both approaches.

Mathematics Grade 8

• Districts with significant change
  • Four out 11 districts had significant change under the jurisdiction-specific approach. This is three fewer than had significant change under the nation-based approach.

• Significantly more inclusive
  • Only one district was significantly more inclusive and was so under both approaches.

• Significantly less inclusive
  • Three districts were significantly less inclusive under the jurisdiction-specific approach. The nation-based approach had three additional districts found to be significantly less inclusive.

Read more about the relation of exclusion and accommodation rates to results.