NAEP Technical DocumentationDevelopment of the Social Studies Indices

For the 2014 social studies assessments (civics, geography, and U.S. history), several indices of policy interest were developed that satisfied both theoretical criteria based on content analysis and empirical criteria based on multivariate statistical techniques. This resulted in the creation of several new reporting elements. The development of the 2014 social studies indices can be summarized in three main steps: 

1. Question selection. A content analysis of the existing 2014 social studies survey questionnaires was conducted to identify sets of questions or items that were theoretically interpretable and meaningful as a conceptual unit, as potential indices to measure specific constructs of interest.

2.  Examination of empirical relationships. Factor analysis was used to explore and verify the empirical properties of the data. Construct validity of the potential indices was evaluated through factorial validity with respect to the survey question responses, and the convergent and discriminant validity of the factor with respect to other factors. If the factor had the expected pattern of relationships and non-relationships, the construct validity of the factor as representing the intended index was supported.   

3. Index scoring. The partial credit item response theory (IRT) model was used to scale the indices. From the IRT model, expected a priori (EAP) scores were generated. Then, the EAP scores were transformed to have a mean of 10 and a standard deviation of 2, and were reported on a scale from 0 – 20.  

Indices of Students' Views of Civics, Geography, and U.S. History

The tables below present the items making up the indices of students' views of civics, geography, and U.S. history, respectively. The indices were designed to measure students' view of their social studies coursework based on their level of agreement with five items. For each of four statements, students were asked to indicate how often they agreed with the statement (never or hardly ever, sometimes, often, or always or almost always). A fifth item asked students to indicate the extent to which they agreed that social studies, civics, geography, or U.S. history was one of their favorite subjects (not at all, a little, or a lot). Statements expressing negative perceptions (e.g., "The U.S. history or social studies work is difficult.") were reverse coded during the scaling process. 
   

Top

Using EAP scores generated from the partial credit IRT model, student responses were converted to a reporting index ranging from 0 to 20, with a mean of 10 and a standard deviation of 2, as described above. The following tables show the IRT parameters for the index of students' views of civics, geography, and U.S. history, respectively.

Top

The tables below show the index values and transformed scale scores for the civics, geography, and U.S. history indices, respectively.

As a reporting aid, cut points were used to divide each scale into low, moderate, and high classifications, such that students with high scores (for example, higher than 11.0 on a scale of 0 to 20) on the index of students' views of U.S. history, generally reported their course material was easy and expressed a strong interest in the topics covered in their courses. Students with low scores (lower than 8.0) on the index of students' views of U.S. history, generally found their course material difficult and indicated little interest in the topics studied.

For each item, response categories were coded as numerical values. The partial credit IRT model was used to scale the indices, and each index value corresponds to a response average. High response averages correspond to high index values, and low response averages correspond to low index values. The index scores were divided into a range of classifications based on the response averages. The cut points for each index were determined based on the average response to the relevant set of items. For instance, for the index of students' views of civics, students were classified as follows:

• Students with index scores associated with an average response less than 2 (index value -0.9127) were classified as low on the index. That is, students who on average responded closest to never or hardly ever on a question with response options never or hardly ever, sometimes, often, and every day or almost every day generally reported that their course material was difficult and indicated little interest in the topics studied.
• Students with index scores associated with an average response greater than or equal to 2 but less than 3 (index value 0.5871) were classified as moderate on the index.
• Finally, students with index scores associated with an average response greater than or equal to 3 were classified as high on the index. That is, students who on average responded closest to often generally reported that their course material was easy and expressed a strong interest in the topics studied in their courses.
  

Printer-friendly Version