NAEP Technical DocumentationDevelopment of the Reading Indices

For the 2015 reading assessment, several indices of policy interest were developed that satisfied both theoretical criteria based on content analysis, and empirical criteria based on multivariate statistical techniques. This resulted in the creation of several new reporting elements. The development of the 2015 reading indices can be summarized in three main steps: 

1. Question selection. A content analysis of the existing subject-specific 2015 reading survey questionnaires was conducted to select sets of the questions that were theoretically interpretable and meaningful as a conceptual unit, as potential indices to measure specific constructs of interest.

2. Examination of empirical relationships. Factor analysis was used to explore and verify the empirical properties of the data. Construct validity of the potential indices was evaluated through factorial validity with respect to the survey question responses, and the convergent and discriminant validity of the factor with respect to other factors. If the factor had the expected pattern of relationships and non-relationships, the construct validity of the factor as representing the intended index was supported.   

3. Index scoring. The partial credit item response theory (IRT) model was used to scale the indices. Scaling of the index items was first conducted to get the item parameters and was based on the marginal maximum likelihood methodologies. After the parameters were estimated, expected a priori (EAP) scores were calculated as the estimate of the index score. Then, the EAP scores were transformed to have a mean of 10 and a standard deviation of 2, and were reported on a scale from 0–20.

Indices were formed for grades 8 and 12 reading, but the grade 4 survey questionnaire did not include a sufficient number of questions that would allow for the creation of indices.  

Index of Students' Views on Reading

The tables below present the items forming the index of students' views on reading at grades 8 and 12, respectively. This index was designed to measure students' interest in and enjoyment of reading and their engagement in reading-related activities. Students were asked to indicate the extent to which they agreed with three statements about reading (strongly disagree, disagree, agree, or strongly agree). Two items asked students how often they engaged in reading-related activities (never or hardly ever, once or twice a month, once or twice a week, or almost every day). 

Index of Students' Computer Use in Reading

The table below presents the items forming the index of students' computer use in reading at grade 12. This index was designed to measure students' use of computers to complete reading-related activities in school. For each of seven items, students were asked to indicate how often they used a computer to do various reading-related activities in their English/language arts class (never or hardly ever, once or twice a month, once or twice a week, or every day or almost every day).

As noted above, partial credit IRT models were estimated for each index, in order to estimate the EAP scores. The following tables show the IRT parameters for the 2015 reading indices.

As a reporting aid, cut points were used to divide the grade 8 and 12 students' views on reading index into low, moderate, and high classifications, and the grade 12 students' computer use in reading index into never, once every few weeks, and once a week or more classifications.

To form these classifications, response categories for each item were first coded as numerical values (e.g., the first response category strongly disagree was coded as 1; the second response category disagree was coded as 2). Note that for all of the indices, each index value corresponds to a response average (e.g., for the grade 8 students' views on reading index, a transformed index score of 5.7 corresponded to an average response of 1, or an average response of strongly disagree). In general, higher response averages correspond to smaller index values, and lower response averages correspond to larger index values. The index scores were divided into a range of classifications based on the response averages. The cut points for each index were determined based on the average response to the relevant set of items. For instance, for the index of students' views on reading, grade 8 students were classified as follows:

• Students with index scores associated with an average response less than the second response category (a transformed index score of 9.0) were classified as low on the index. That is, students who on average responded less than disagree on a question with response options strongly disagree, disagree, agree, and strongly agree generally reported that reading was not one of their favorite activities and that reading was not enjoyable to them, and were classified as low on the index.
  
• Students with index scores associated with an average response greater than or equal to the second response category but less than the third response category (a transformed index score of 11.3) were classified as moderate on the index.
• Finally, students with index scores associated with an average response greater than or equal to the third response category were classified as high on the index. That is, students who on average responded agree or higher generally reported that reading was one of their favorite activities and that reading was enjoyable to them, and were classified as high on the index.
  

Printer-friendly Version