NAEP Technical Documentation Development of the 2017 Grade 8 Reading Indices

For the 2017 grade 8 reading assessment, several indices of policy interest were developed that satisfied both theoretical criteria based on content analysis, and empirical criteria based on multivariate statistical techniques. This resulted in the creation of several new reporting elements. The development of the 2017 grade 8 reading indices can be summarized in three main steps:

1. Question development. New sets of contextual items, such as those exploring students’ persistence in learning and their enjoyment of complex problems, were developed and included in the reading student questionnaire. Through content analysis as part of the item development process, sets of items that were theoretically interpretable and meaningful as a conceptual unit were included as potential indices to measure specific constructs of interest.

2. Examination of empirical relationships. Factor analysis was used to explore and verify the empirical properties of the data. Construct validity of the potential indices was evaluated through factorial validity with respect to the survey question responses, and the convergent and discriminant validity of the factor with respect to other factors. If the factor had the expected pattern of relationships and non-relationships, the construct validity of the factor as representing the intended index was supported.

3. Index scoring. The partial credit item response theory (IRT) model was used to scale the indices. Scaling of the index items was first conducted to get the item parameters and was based on the marginal maximum likelihood methodologies. After the parameters were estimated, expected a posteriori (EAP) scores were calculated as the estimate of the index score. Then, the EAP scores were transformed to have a mean of 10 and a standard deviation of 2, and were reported on a scale from 0–20.

Index of Students' Persistence in Learning

The table below presents the items forming the index of students' persistence in learning. This index was designed to measure students' tendency to persevere and work hard in the face of challenges. Grade 8 students were asked to indicate how much each of the four statement items described a person like them (not at all like me, a little bit like me, somewhat like me, quite a bit like me, or very much like me).

Items in the index of eighth-grade students' persistence in learning, grade 8 reading combined national and state assessment: 2017

How much does each of the following statements describe a person like you? Select one answer choice on each row.
Response categories
Item			Not at all like me	A little bit like me	Somewhat like me	Quite a bit like me	Very much like me
B034901	a.	I finish whatever I begin.	A	B	C	D	E
B034902	b.	I try very hard even after making mistakes.	A	B	C	D	E
B034903	c.	I keep working hard even when I feel like quitting.	A	B	C	D	E
B034904	d.	I keep trying to improve myself, even when it takes a long time to get there.	A	B	C	D	E
SOURCE: U.S. Department of Education, Institute of Education Sciences, National Center for Education Statistics, National Assessment of Educational Progress (NAEP), 2017 Reading Assessment.

Index of Students' Academic Self-Discipline

The table below presents the items forming the index of students' academic self-discipline. This index was designed to measure students' tendency to pay attention and stay on task when learning. Grade 8 students were asked to indicate how often they have done each of the four statement items (never or hardly ever, less than half of the time, about half of the time, more than half of the time, or all or almost all of the time).

Items in the index of eighth-grade students' academic self-discipline, grade 8 reading combined national and state assessment: 2017

In this school year, how often have you done each of the following? Select one answer choice on each row.
Response categories
Item			Never or hardly ever	Less than half of the time	About half of the time	More than half of the time	All or almost all of the time
B035001	a.	I started working on assignments right away rather than waiting until the last minute.	A	B	C	D	E
B035002	b.	I paid attention and resisted distractions.	A	B	C	D	E
B035003	c.	I stayed on task without reminders from my teacher.	A	B	C	D	E
B035004	d.	I paid attention in class even when I was not interested.	A	B	C	D	E
SOURCE: U.S. Department of Education, Institute of Education Sciences, National Center for Education Statistics, National Assessment of Educational Progress (NAEP), 2017 Reading Assessment.

Index of Students' Enjoyment of Complex Problems

The table below presents the items forming the index of students' enjoyment of complex problems. This index was designed to measure students' enjoyment of problems and activities that challenge them to think. Grade 8 students were asked to indicate how much each of the four statement items described a person like them (not at all like me, a little bit like me, somewhat like me, quite a bit like me, or very much like me).

Items in the index of eighth-grade students' enjoyment of complex problems, grade 8 reading combined national and state assessment: 2017

How much does each of the following statements describe a person like you? Select one answer choice on each row.
Response categories
Item			Not at all like me	A little bit like me	Somewhat like me	Quite a bit like me	Very much like me
B035101	a.	I like complex problems more than easy problems.	A	B	C	D	E
B035102	b.	I like activities that challenge my thinking abilities.	A	B	C	D	E
B035103	c.	I enjoy situations where I will have to think about something.	A	B	C	D	E
B035104	d.	I enjoy thinking about new solutions to problems.	A	B	C	D	E
SOURCE: U.S. Department of Education, Institute of Education Sciences, National Center for Education Statistics, National Assessment of Educational Progress (NAEP), 2017 Reading Assessment.

Index of Students’ Confidence in Reading Knowledge and Skills

The table below presents the items forming the index of students' confidence in their reading knowledge and skills at grade 8. This index was designed to measure students' belief in their abilities to do various reading-related tasks. Grade 8 students were asked to indicate their confidence in doing the task described in each of ten items (I definitely can’t, I probably can’t, maybe, I probably can, or I definitely can).

Items in the index of eighth-grade students' confidence in reading knowledge and skills, grade 8 reading combined national and state assessment: 2017

Do you think you would be able to do each of the following when reading? Select one answer choice on each row.
Response categories
Item			I definitely can't	I probably can't	Maybe	I probably can	I definitely can
R849601	a.	Figure out the meaning of a word you don't know by using other words in the text	A	B	C	D	E
R849602	b.	Explain the meaning of something you have read	A	B	C	D	E
R849603	c.	Figure out the main idea of a text	A	B	C	D	E
R849604	d.	Find text in a reading passage to help you answer a question on a test	A	B	C	D	E
R849605	e.	Recognize when you don't understand something you are reading	A	B	C	D	E
Do you think you would be able to do each of the following when reading? Select one answer choice on each row.
Response categories
Item			I definitely can't	I probably can't	Maybe	I probably can	I definitely can
R850301	a.	Recognize the difference between fact and opinion in a text	A	B	C	D	E
R850302	b.	Judge the reliability of sources (for example, how a website might be biased or inaccurate)	A	B	C	D	E
R850303	c.	Critique an author's craft or technique	A	B	C	D	E
R850304	d.	Use evidence from a text to support my answer	A	B	C	D	E
R850305	e.	Identify the author's perspective in a persuasive text	A	B	C	D	E
SOURCE: U.S. Department of Education, Institute of Education Sciences, National Center for Education Statistics, National Assessment of Educational Progress (NAEP), 2017 Reading Assessment.

Index of Students’ Interest/Enjoyment in Reading

The table below presents the items forming the index of students' interest/enjoyment in reading. This index was designed to measure students' interest in and enjoyment of reading, and their views of the importance of reading. Grade 8 students were asked to indicate how much each of the four statement items described a person like them (not at all like me, a little bit like me, somewhat like me, quite a bit like me, or exactly like me).

Items in the index of eighth-grade students' interest/enjoyment in reading, grade 8 reading combined national and state assessment: 2017

How much does each of the following statements describe a person like you? Select one answer choice on each row.
Response categories
Item			Not at all like me	A little bit like me	Somewhat like me	Quite a bit like me	Exactly like me
R849801	a.	Reading is one of my favorite activities.	A	B	C	D	E
R849802	b.	I like talking about books with other people.	A	B	C	D	E
R849803	c.	I think reading is important.	A	B	C	D	E
R849804	d.	I enjoy going to a bookstore or a library.	A	B	C	D	E
SOURCE: U.S. Department of Education, Institute of Education Sciences, National Center for Education Statistics, National Assessment of Educational Progress (NAEP), 2017 Reading Assessment.

For each index item, the response categories were scored as numerical values (e.g., for an item with five response categories, category A was scored as 1, B was scored as 2, C was scored as 3, D was scored as 4, and E was scored as 5). For the 2017 grade 8 reading indices, item response categories were collapsed; scores for a five-category item thus became 1, 2, 3, and 4 after collapsing. The table below describes the treatment of the index items.

Recoding of polytomous items for scaling, grade 8 reading combined national and state assessment, by index: 2017

Item	Index	Reason for decision	Disposition
B034901	Persistence in learning	To improve model-data fit	Collapse categories: 1,2,3,4,5 becomes 1,1,2,3,4
B034902	Persistence in learning	To improve model-data fit	Collapse categories: 1,2,3,4,5 becomes 1,1,2,3,4
B034903	Persistence in learning	To improve model-data fit	Collapse categories: 1,2,3,4,5 becomes 1,1,2,3,4
B034904	Persistence in learning	To improve model-data fit	Collapse categories: 1,2,3,4,5 becomes 1,1,2,3,4
B035001	Academic self-discipline	To improve model-data fit	Collapse categories: 1,2,3,4,5 becomes 1,1,2,3,4
B035002	Academic self-discipline	To improve model-data fit	Collapse categories: 1,2,3,4,5 becomes 1,1,2,3,4
B035003	Academic self-discipline	To improve model-data fit	Collapse categories: 1,2,3,4,5 becomes 1,1,2,3,4
B035004	Academic self-discipline	To improve model-data fit	Collapse categories: 1,2,3,4,5 becomes 1,1,2,3,4
B035101	Enjoyment of complex problems	To improve model-data fit	Collapse categories: 1,2,3,4,5 becomes 1,1,2,3,4
B035102	Enjoyment of complex problems	To improve model-data fit	Collapse categories: 1,2,3,4,5 becomes 1,1,2,3,4
B035103	Enjoyment of complex problems	To improve model-data fit	Collapse categories: 1,2,3,4,5 becomes 1,1,2,3,4
B035104	Enjoyment of complex problems	To improve model-data fit	Collapse categories: 1,2,3,4,5 becomes 1,1,2,3,4
R849601	Confidence in reading knowledge and skills	To improve model-data fit	Collapse categories: 1,2,3,4,5 becomes 1,1,2,3,4
R849602	Confidence in reading knowledge and skills	To improve model-data fit	Collapse categories: 1,2,3,4,5 becomes 1,1,2,3,4
R849603	Confidence in reading knowledge and skills	To improve model-data fit	Collapse categories: 1,2,3,4,5 becomes 1,1,2,3,4
R849604	Confidence in reading knowledge and skills	To improve model-data fit	Collapse categories: 1,2,3,4,5 becomes 1,1,2,3,4
R849605	Confidence in reading knowledge and skills	To improve model-data fit	Collapse categories: 1,2,3,4,5 becomes 1,1,2,3,4
R850301	Confidence in reading knowledge and skills	To improve model-data fit	Collapse categories: 1,2,3,4,5 becomes 1,1,2,3,4
R850302	Confidence in reading knowledge and skills	To improve model-data fit	Collapse categories: 1,2,3,4,5 becomes 1,1,2,3,4
R850303	Confidence in reading knowledge and skills	To improve model-data fit	Collapse categories: 1,2,3,4,5 becomes 1,1,2,3,4
R850304	Confidence in reading knowledge and skills	To improve model-data fit	Collapse categories: 1,2,3,4,5 becomes 1,1,2,3,4
R850305	Confidence in reading knowledge and skills	To improve model-data fit	Collapse categories: 1,2,3,4,5 becomes 1,1,2,3,4
R849801	Interest/enjoyment in reading	To improve model-data fit	Collapse categories: 1,2,3,4,5 becomes 1,1,2,3,4
R849802	Interest/enjoyment in reading	To improve model-data fit	Collapse categories: 1,2,3,4,5 becomes 1,1,2,3,4
R849803	Interest/enjoyment in reading	To improve model-data fit	Collapse categories: 1,2,3,4,5 becomes 1,1,2,3,4
R849804	Interest/enjoyment in reading	To improve model-data fit	Collapse categories: 1,2,3,4,5 becomes 1,1,2,3,4
SOURCE: U.S. Department of Education, Institute of Education Sciences, National Center for Education Statistics, National Assessment of Educational Progress (NAEP), 2017 Reading Assessment.

The partial credit IRT model was used to scale the indices. Scaling of the index items was first conducted to get the item parameters and was based on the marginal maximum likelihood methodologies. The following tables show the IRT parameters for the 2017 grade 8 reading indices.

IRT parameters for items in the index of eighth-grade students' persistence in learning, grade 8 reading combined national and state assessment: 2017

Item	bj	dj1	dj2	dj3	dj4	dj5
B034901	-0.39	0.99	0.05	-1.04	†	†
B034902	-0.63	0.78	0.06	-0.83	†	†
B034903	-0.42	0.73	0.02	-0.75	†	†
B034904	-0.84	0.59	0.04	-0.63	†	†
† Not applicable. NOTE: The number of dji parameters is one less than the number of categories for the item. For items scaled with the partial credit model, parameters a and c are not estimated. The a parameter value is exactly one and the c parameter is exactly zero. For item j, bj represents a location parameter related to item difficulty, and dji represents the category threshold parameter for category i of item j; dji may not sum to zero because of rounding. SOURCE: U.S. Department of Education, Institute of Education Sciences, National Center for Education Statistics, National Assessment of Educational Progress (NAEP), 2017 Reading Assessment.

IRT parameters for items in the index of eighth-grade students' academic self-discipline, grade 8 reading combined national and state assessment: 2017

Item	bj	dj1	dj2	dj3	dj4	dj5
B035001	0.03	0.81	-0.05	-0.76	†	†
B035002	-0.17	0.92	0.08	-1.00	†	†
B035003	-0.44	0.73	0.03	-0.75	†	†
B035004	-0.28	0.72	0.04	-0.76	†	†
† Not applicable. NOTE: The number of dji parameters is one less than the number of categories for the item. For items scaled with the partial credit model, parameters a and c are not estimated. The a parameter value is exactly one and the c parameter is exactly zero. For item j, bj represents a location parameter related to item difficulty, and dji represents the category threshold parameter for category i of item j; dji may not sum to zero because of rounding. SOURCE: U.S. Department of Education, Institute of Education Sciences, National Center for Education Statistics, National Assessment of Educational Progress (NAEP), 2017 Reading Assessment.

IRT parameters for items in the index of eighth-grade students' enjoyment of complex problems, grade 8 reading combined national and state assessment: 2017

Item	bj	dj1	dj2	dj3	dj4	dj5
B035101	0.46	0.79	-0.01	-0.78	†	†
B035102	-0.08	0.76	-0.01	-0.75	†	†
B035103	-0.03	0.78	0.00	-0.78	†	†
B035104	-0.09	0.67	-0.01	-0.66	†	†
† Not applicable. NOTE: The number of dji parameters is one less than the number of categories for the item. For items scaled with the partial credit model, parameters a and c are not estimated. The a parameter value is exactly one and the c parameter is exactly zero. For item j, bj represents a location parameter related to item difficulty, and dji represents the category threshold parameter for category i of item j; dji may not sum to zero because of rounding. SOURCE: U.S. Department of Education, Institute of Education Sciences, National Center for Education Statistics, National Assessment of Educational Progress (NAEP), 2017 Reading Assessment.

IRT parameters for items in the index of eighth-grade students' confidence in reading knowledge and skills, grade 8 reading combined national and state assessment: 2017

Item	bj	dj1	dj2	dj3	dj4	dj5
R849601	-0.80	1.09	0.11	-1.21	†	†
R849602	-0.85	1.07	0.08	-1.15	†	†
R849603	-0.99	1.02	0.06	-1.09	†	†
R849604	-1.05	0.97	0.02	-0.99	†	†
R849605	-0.95	0.92	-0.13	-0.79	†	†
R850301	-1.29	0.81	-0.16	-0.65	†	†
R850302	-0.61	1.13	-0.01	-1.11	†	†
R850303	-0.21	1.18	-0.07	-1.12	†	†
R850304	-1.17	0.90	-0.01	-0.89	†	†
R850305	-0.86	1.03	-0.06	-0.98	†	†
† Not applicable. NOTE: The number of dji parameters is one less than the number of categories for the item. For items scaled with the partial credit model, parameters a and c are not estimated. The a parameter value is exactly one and the c parameter is exactly zero. For item j, bj represents a location parameter related to item difficulty, and dji represents the category threshold parameter for category i of item j; dji may not sum to zero because of rounding. SOURCE: U.S. Department of Education, Institute of Education Sciences, National Center for Education Statistics, National Assessment of Educational Progress (NAEP), 2017 Reading Assessment.

IRT parameters for items in the index of eighth-grade students' interest/enjoyment in reading, grade 8 reading combined national and state assessment: 2017

Item	bj	dj1	dj2	dj3	dj4	dj5
R849801	0.55	0.37	0.02	-0.39	†	†
R849802	0.76	0.27	0.03	-0.31	†	†
R849803	-0.35	0.60	0.00	-0.59	†	†
R849804	0.37	0.17	-0.03	-0.14	†	†
† Not applicable. NOTE: The number of dji parameters is one less than the number of categories for the item. For items scaled with the partial credit model, parameters a and c are not estimated. The a parameter value is exactly one and the c parameter is exactly zero. For item j, bj represents a location parameter related to item difficulty, and dji represents the category threshold parameter for category i of item j; dji may not sum to zero because of rounding. SOURCE: U.S. Department of Education, Institute of Education Sciences, National Center for Education Statistics, National Assessment of Educational Progress (NAEP), 2017 Reading Assessment.

After the parameters were estimated, EAP scores were calculated as the estimate of the index score. Then, the EAP scores were transformed to have a mean of 10 and a standard deviation of 2, and were reported on a scale from 0–20 (transformed scale score = 2 * (index value) + 10). The tables below show the response averages, index values, and transformed scale scores for each of the five grade 8 reading indices. Note that response averages represent raw response averages after collapsing. Each response average corresponds to one index value and transformed score.

As a reporting aid, index scores were divided into a range of categories or classifications (e.g., low, moderate, high). The cut points selected to divide the index scores into meaningful categories were based on the distributions of the EAP scores; there are no theoretical derivations behind the selection of the cut points. For example, for the index of students’ persistence in learning, index scores were divided into low, moderate, and high classifications, such that grade 8 students with high index scores (i.e., a transformed score of 9.9 or higher on a scale of 0 to 20) generally reported that statements about persevering and working hard in the face of challenges described a person like them quite a bit or very much. Grade 8 students with low scores on the index (i.e., lower than 8.1) generally reported that these statements described a person like them not at all or a little bit. Higher index values and transformed scores correspond to higher response averages, and lower index values and transformed scores correspond to lower response averages.

As an example, for the index of persistence in learning, grade 8 students were classified as follows:

• Students with index scores associated with a response average less than 2 (index value -0.93928) were classified as low on the index. That is, students who on average responded closest to not at all like me on a question with the five response options not at all like me, a little bit like me, somewhat like me, quite a bit like me, and very much like me were classified as having a low level of persistence in learning.
• Students with index scores associated with a response average greater than or equal to 2 to less than 3 (index value -0.07623) were classified as having a moderate level of persistence in learning.
• Finally, students with index scores associated with a response average greater than or equal to 3 were classified as high on the index. That is, students who on average responded closest to very much like me were classified as having a high level of persistence in learning.

Response averages, index values, and transformed scores for the index of eighth-grade students' persistence in learning, grade 8 reading combined national and state assessment, by classification: 2017

Classification	Response average	Index value	Transformed score	Percentage of students
Low	1.00	-2.11245	5.8	2
	1.25	-1.71559	6.6	2
	1.50	-1.41434	7.2	3
	1.75	-1.16353	7.7	4
Moderate	2.00	-0.93928	8.1	6
	2.25	-0.72765	8.5	6
	2.50	-0.51887	9.0	8
	2.75	-0.30465	9.4	9
High	3.00	-0.07623	9.9	12
	3.25	0.17795	10.4	11
	3.50	0.47622	11.0	12
	3.75	0.85290	11.7	12
	4.00	1.38004	12.8	14
NOTE: Detail may not sum to totals because of rounding. SOURCE: U.S. Department of Education, Institute of Education Sciences, National Center for Education Statistics, National Assessment of Educational Progress (NAEP), 2017 Reading Assessment.

Response averages, index values, and transformed scores for the index of eighth-grade students' academic self-discipline, grade 8 reading combined national and state assessment, by classification: 2017

Classification	Response average	Index value	Transformed score	Percentage of students
Low	1.00	-1.87416	6.3	3
	1.25	-1.44280	7.1	3
	1.50	-1.12152	7.8	4
	1.75	-0.85750	8.3	6
Moderate	2.00	-0.62414	8.8	9
	2.25	-0.40651	9.2	9
	2.50	-0.19465	9.6	10
	2.75	0.01946	10.0	10
High	3.00	0.24397	10.5	11
	3.25	0.48937	11.0	10
	3.50	0.77201	11.5	10
	3.75	1.12214	12.2	7
	4.00	1.60225	13.2	7
NOTE: Detail may not sum to totals because of rounding. SOURCE: U.S. Department of Education, Institute of Education Sciences, National Center for Education Statistics, National Assessment of Educational Progress (NAEP), 2017 Reading Assessment.

Response averages, index values, and transformed scores for the index of eighth-grade students' enjoyment of complex problems, grade 8 reading combined national and state assessment, by classification: 2017

Classification	Response average	Index value	Transformed score	Percentage of students
Low	1.00	-1.66639	6.7	11
	1.25	-1.20551	7.6	6
	1.50	-0.87114	8.3	6
	1.75	-0.60216	8.8	7
Moderate	2.00	-0.36876	9.3	10
	2.25	-0.15445	9.7	7
	2.50	0.05151	10.1	8
	2.75	0.25761	10.5	8
High	3.00	0.47228	10.9	9
	3.25	0.70594	11.4	6
	3.50	0.97435	12.0	6
	3.75	1.30569	12.6	5
	4.00	1.75653	13.5	8
NOTE: Detail may not sum to totals because of rounding. SOURCE: U.S. Department of Education, Institute of Education Sciences, National Center for Education Statistics, National Assessment of Educational Progress (NAEP), 2017 Reading Assessment.

Response averages, index values, and transformed scores for the index of eighth-grade students' confidence in reading knowledge and skills, grade 8 reading combined national and state assessment, by classification: 2017

Classification	Response average	Index value	Transformed score	Percentage of students
Low	1.00	-2.97129	4.1	1
	1.10	-2.68907	4.6	#
	1.20	-2.45942	5.1	#
	1.30	-2.26911	5.5	#
	1.40	-2.10496	5.8	#
	1.50	-1.95868	6.1	#
	1.60	-1.82500	6.4	#
	1.70	-1.70060	6.6	1
	1.80	-1.58303	6.8	1
	1.90	-1.47066	7.1	1
Moderate	2.00	-1.36210	7.3	3
	2.10	-1.25637	7.5	1
	2.20	-1.15262	7.7	2
	2.30	-1.05005	7.9	2
	2.40	-0.94803	8.1	2
	2.50	-0.84587	8.3	3
	2.60	-0.74298	8.5	3
	2.70	-0.63865	8.7	3
	2.80	-0.53222	8.9	4
	2.90	-0.42291	9.2	4
High	3.00	-0.30984	9.4	7
	3.10	-0.19191	9.6	5
	3.20	-0.06780	9.9	6
	3.30	0.06422	10.1	6
	3.40	0.20652	10.4	6
	3.50	0.36247	10.7	7
	3.60	0.53721	11.1	7
	3.70	0.73897	11.5	7
	3.80	0.98222	12.0	6
	3.90	1.29469	12.6	5
	4.00	1.73434	13.5	8
# Rounds to zero. NOTE: Detail may not sum to totals because of rounding. SOURCE: U.S. Department of Education, Institute of Education Sciences, National Center for Education Statistics, National Assessment of Educational Progress (NAEP), 2017 Reading Assessment.

Response averages, index values, and transformed scores for the index of eighth-grade students' interest/enjoyment in reading, grade 8 reading combined national and state assessment, by classification: 2017

Classification	Response average	Index value	Transformed score	Percentage of students
Low	1.00	-1.39698	7.2	14
	1.25	-0.88775	8.2	12
	1.50	-0.53366	8.9	10
	1.75	-0.26245	9.5	10
Moderate	2.00	-0.03928	9.9	9
	2.25	0.15517	10.3	6
	2.50	0.33359	10.7	7
	2.75	0.50589	11.0	5
High	3.00	0.68154	11.4	5
	3.25	0.87176	11.7	5
	3.50	1.09314	12.2	4
	3.75	1.37549	12.8	4
	4.00	1.78001	13.6	8
NOTE: Detail may not sum to totals because of rounding. SOURCE: U.S. Department of Education, Institute of Education Sciences, National Center for Education Statistics, National Assessment of Educational Progress (NAEP), 2017 Reading Assessment.

Printer-friendly Version