NAEP Technical DocumentationDevelopment of the Science Indices

For the 2015 science assessment, several indices of policy interest were developed that satisfied both theoretical criteria based on content analysis, and empirical criteria based on multivariate statistical techniques. This resulted in the creation of several new reporting elements. The development of the 2015 science indices can be summarized in three main steps:

1. Question selection. A content analysis of the existing subject-specific 2015 science survey questionnaires was conducted to select sets of the questions that were theoretically interpretable and meaningful as a conceptual unit, as potential indices to measure specific constructs of interest.

2. Examination of empirical relationships. Factor analysis was used to explore and verify the empirical properties of the data. Construct validity of the potential indices was evaluated through factorial validity with respect to the survey question responses, and the convergent and discriminant validity of the factor with respect to other factors. If the factor had the expected pattern of relationships and non-relationships, the construct validity of the factor as representing the intended index was supported.

3. Index scoring. The partial credit item response theory (IRT) model was used to scale the indices. Scaling of the index items was first conducted to get the item parameters and was based on the marginal maximum likelihood methodologies. After the parameters were estimated, expected a priori (EAP) scores were calculated as the estimate of the index score. Then, the EAP scores were transformed to have a mean of 10 and a standard deviation of 2, and were reported on a scale from 0–20.

Indices were formed for grades 8 and 12 science, but the grade 4 survey questionnaire did not include a sufficient number of questions that would allow for the creation of indices.

Index of Students' Views on Science

The tables below present the items forming the index of students’ views on science at grades 8 and 12, respectively. This index was designed to measure students’ interest in and enjoyment of science. At grades 8 and 12, students were asked to indicate the extent to which they agreed with five statements about science (strongly disagree, disagree, agree, or strongly agree).

Items in the index of eighth-grade students' views on science, grade 8 science combined national and state assessment: 2015
Please indicate how much you DISAGREE or AGREE with the following statements about science. Fill in one oval on each line.
Response categories
Item			Strongly disagree	Disagree	Agree	Strongly agree
¹Item is reverse coded. SOURCE: U.S. Department of Education, Institute of Education Sciences, National Center for Education Statistics, National Assessment of Educational Progress (NAEP), 2015 Science Assessment.
K820501	a.	I do science-related activities that are not for schoolwork.	A	B	C	D
K816604	b.	I like science.	A	B	C	D
K816605	c.	Science is one of my favorite subjects.	A	B	C	D
K816606	d.	I take science only because I have to.¹	A	B	C	D
K816609	e.	I would like a job that involves using science.	A	B	C	D

Items in the index of twelfth-grade students' views on science, grade 12 science national assessment: 2015
Please indicate how much you DISAGREE or AGREE with the following statements about science. Fill in one oval on each line.
Response categories
Item			Strongly disagree	Disagree	Agree	Strongly agree
¹Item is reverse coded. SOURCE: U.S. Department of Education, Institute of Education Sciences, National Center for Education Statistics, National Assessment of Educational Progress (NAEP), 2015 Science Assessment.
K820501	a.	I do science-related activities that are not for schoolwork.	A	B	C	D
K816604	b.	I like science.	A	B	C	D
K816605	c.	Science is one of my favorite subjects.	A	B	C	D
K816606	d.	I take science only because I have to.¹	A	B	C	D
K816609	e.	I would like a job that involves using science.	A	B	C	D

As noted above, partial credit IRT models were estimated for each index, in order to estimate the EAP scores. The following tables show the IRT parameters for the 2015 science indices.

IRT parameters for the index of eighth-grade students' views on science, grade 8 science combined national and state assessment, by item: 2015
Item	b_j	d_j1	d_j2	d_j3	d_j4	d_j5
K816602	0.64	1.45	0.04	-1.49	†	†
K816604	-0.48	0.91	0.49	-1.40	†	†
K816605	-0.09	0.96	0.06	-1.02	†	†
K816606	-0.11	0.96	0.08	-1.04	†	†
K816609	0.18	0.99	-0.09	-0.90	†	†
† Not applicable. NOTE: The number of d_ji parameters is one less than the number of categories for the item. For items scaled with the partial credit model, parameters a and c are not estimated. The a parameter value is exactly one, and the c parameter is exactly zero. For item j, b_j represents a location parameter related to item difficulty, and d_ji represents the category threshold parameter for category i of item j; d_ji may not sum to zero because of rounding. SOURCE: U.S. Department of Education, Institute of Education Sciences, National Center for Education Statistics, National Assessment of Educational Progress (NAEP), 2015 Science Assessment.

IRT parameters for the index of twelfth-grade students' views on science, grade 12 science national assessment, by item: 2015
Item	b_j	d_j1	d_j2	d_j3	d_j4	d_j5
K816602	0.56	1.36	0.13	-1.49	†	†
K816604	-0.39	1.06	0.48	-1.54	†	†
K816605	0.10	1.10	0.01	-1.10	†	†
K816606	0.01	1.19	0.08	-1.26	†	†
K816609	0.17	0.96	-0.07	-0.90	†	†
† Not applicable. NOTE: The number of d_ji parameters is one less than the number of categories for the item. For items scaled with the partial credit model, parameters a and c are not estimated. The a parameter value is exactly one, and the c parameter is exactly zero. For item j, b_j represents a location parameter related to item difficulty, and d_ji represents the category threshold parameter for category i of item j; d_ji may not sum to zero because of rounding. SOURCE: U.S. Department of Education, Institute of Education Sciences, National Center for Education Statistics, National Assessment of Educational Progress (NAEP), 2015 Science Assessment.

The EAP scores generated from the partial credit IRT models were transformed to reporting index scores ranging from 0 to 20, with a mean of 10 and a standard deviation of 2, as noted above.

As a reporting aid, cut points were used to divide the grade 8 and 12 students’ views on science index into low, moderate, and high classifications.

To form these classifications, response categories for each item were first coded as numerical values (e.g., the first response category strongly disagree was coded as 1; the second response category disagree was coded as 2). Note that for all of the indices, each index value corresponds to a response average (e.g., for the grade 8 students’ views on science index, a transformed index score of 6.0 corresponded to an average response of 1, or an average response of strongly disagree). In general, higher response averages correspond to smaller index values, and lower response averages correspond to larger index values. The index scores were divided into a range of classifications based on the response averages. The cut points for each index were determined based on the average response to the relevant set of items. For instance, for the index of students' views of science, grade 8 students were classified as follows:

Students with index scores associated with an average response less than the second response category (a transformed index score of 8.9) were classified as low on the index. That is, students who on average responded less than disagree on a question with response options strongly disagree, disagree, agree, and strongly agree generally reported that they did not like science and that science was not one of their favorite subjects, and were classified as low on the index.
Students with index scores associated with an average response greater than or equal to the second response category but less than the third response category (a transformed index score of 11.1) were classified as moderate on the index.
Finally, students with index scores associated with an average response greater than or equal to the third response category were classified as high on the index. That is, students who on average responded agree or higher generally reported that they liked science and that science was one of their favorite subjects, and were classified as high on the index.

The tables below show the index values, transformed scale scores, and score classifications for the science indices.

Index values and transformed scores for the index of eighth-grade students' views on science, grade 8 science combined national and state assessment: 2015
Classification	Index value	Transformed score	Percentage of students
Low	-2.01034	6.0	3
	-1.59841	6.8	3
	-1.28468	7.4	3
	-1.02102	8.0	5
	-0.78348	8.4	6
Moderate	-0.55945	8.9	8
	-0.34176	9.3	9
	-0.12605	9.8	11
	0.09066	10.2	12
	0.31108	10.6	12
High	0.53858	11.1	8
	0.77826	11.6	6
	1.03830	12.1	6
	1.33206	12.7	4
	1.68228	13.4	3
	2.12904	14.3	1
NOTE: Detail may not sum to totals because of rounding. SOURCE: U.S. Department of Education, Institute of Education Sciences, National Center for Education Statistics, National Assessment of Educational Progress (NAEP), 2015 Science Assessment.

Index values and transformed scores for the index of twelfth-grade students' views on science, grade 12 science national assessment: 2015
Classification	Index value	Transformed score	Percentage of students
Low	-2.01575	6.0	5
	-1.59562	6.8	4
	-1.27233	7.5	4
	-0.99874	8.0	6
	-0.75136	8.5	7
Moderate	-0.51777	9.0	8
	-0.29093	9.4	9
	-0.06659	9.9	10
	0.15825	10.3	9
	0.38629	10.8	10
High	0.62075	11.2	8
	0.86621	11.7	5
	1.12972	12.3	5
	1.42293	12.9	4
	1.76644	13.5	4
	2.19769	14.4	3
NOTE: Detail may not sum to totals because of rounding. SOURCE: U.S. Department of Education, Institute of Education Sciences, National Center for Education Statistics, National Assessment of Educational Progress (NAEP), 2015 Science Assessment.

Last updated 01 November 2021 (ML)

Printer-friendly Version