Table of Contents | Search Technical Documentation | References

NAEP Analysis and Scaling → Initial Activities → Constructed-Response Interrater Reliability → Score Range and Percentage of Exact Agreement for the Constructed-Response Items Used in Scaling, Grade 8 Reading Combined National and State Assessment, by Block and Item: 2017

NAEP Technical DocumentationScore Range and Percentage of Exact Agreement for the Constructed-Response Items Used in Scaling, Grade 8 Reading Combined National and State Assessment, by Block and Item: 2017

Score range and percentage of exact agreement for the constructed-response items used in scaling, grade 8 reading combined national and state assessment, by block and item: 2017
Block	Item	Score range	Sample size	Percentage of exact agreement
R1	R059604	1-2	1,100	94
	R059605	1-3	1,000	81
	R059606	1-3	1,100	85
	R059607	1-3	1,000	89
	R059610	1-4	1,100	81
R2	R069801	1-3	1,100	95
	R069804	1-3	1,100	81
	R069806	1-4	1,000	77
	R069810	1-3	1,000	82
R3	R060103	1-2	1,100	86
	R060105	1-2	1,100	94
	R060108	1-4	1,000	76
R4	R080005	1-4	1,000	80
	R080007	1-2	1,000	89
	R080009	1-3	900	83
	R080010	1-3	900	76
R5	R079504	1-3	1,100	85
	R079507	1-4	1,100	76
	R079508	1-2	1,100	91
	R079510	1-3	1,100	77
R6	R058602	1-3	1,100	83
	R058606	1-3	1,100	84
	R058608	1-4	1,000	77
R7	R058805	1-3	1,100	83
	R058807	1-4	1,100	75
	R058810	1-3	1,100	80
R8	R079703	1-3	1,100	87
	R079706	1-3	1,100	94
	R079707	1-4	1,100	92
	R079709	1-3	1,000	80
R9	R060503	1-3	1,100	88
	R060505	1-4	1,100	78
	R060507	1-3	1,100	77
	R060508	1-3	1,000	82
	R060509	1-3	1,000	80
R10	R060603	1-3	1,000	81
	R060605	1-3	1,100	86
	R060606	1-4	1,100	81
	R060607	1-3	1,100	79
	R060608	1-3	1,100	83
R11	R0Q4103	1-3	1,100	88
	R0Q4104	1-3	1,100	82
	R0Q4106	1-3	1,000	76
	R0Q4107	1-4	900	78
R12	R061003	1-3	1,100	83
	R061004	1-2	1,100	92
	R061006	1-2	1,000	91
	R061007	1-4	900	73
	R061010	1-3	900	82
R13	R069705	1-2	1,100	88
	R069706	1-3	1,100	84
	R069707	1-4	1,000	77
	R069708	1-3	1,000	72
NOTE: Special codes assigned to student responses including omitted, off-task, and illegible were included in the calculation of the percentage of exact agreement measure. SOURCE: U.S. Department of Education, Institute of Education Sciences, National Center for Education Statistics, National Assessment of Educational Progress (NAEP), 2017 Reading Assessment.

Score range, Cohen's Kappa, and intraclass correlation for the constructed-response items used in scaling, grade 8 reading combined national and state assessment, by block and item: 2017
Block	Item	Score range	Sample size	Cohen’s Kappa	Intraclass correlation
R1	R059604	1-2	1,100	.78	†
	R059605	1-3	1,000	.69	.77
	R059606	1-3	1,100	.76	.85
	R059607	1-3	1,000	.82	.88
	R059610	1-4	1,100	.74	.87
R2	R069801	1-3	1,100	.89	.91
	R069804	1-3	1,000	.67	.77
	R069806	1-4	1,000	.66	.80
	R069810	1-3	1,000	.74	.82
R3	R060103	1-2	1,100	.67	†
	R060105	1-2	1,100	.88	†
	R060108	1-4	1,000	.68	.82
R4	R080005	1-4	1,000	.72	.89
	R080007	1-2	1,000	.77	†
	R080009	1-3	900	.71	.80
	R080010	1-3	900	.61	.72
R5	R079504	1-3	1,100	.77	.87
	R079507	1-4	1,100	.67	.85
	R079508	1-2	1,100	.73	†
	R079510	1-3	1,100	.65	.77
R6	R058602	1-3	1,100	.73	.82
	R058606	1-3	1,100	.75	.78
	R058608	1-4	1,000	.67	.84
R7	R058805	1-3	1,100	.71	.78
	R058807	1-4	1,100	.62	.78
	R058810	1-3	1,100	.67	.78
R8	R079703	1-3	1,100	.77	.80
	R079706	1-3	1,100	.87	.94
	R079707	1-4	1,100	.88	.95
	R079709	1-3	1,000	.65	.74
R9	R060503	1-3	1,100	.81	.88
	R060505	1-4	1,000	.69	.86
	R060507	1-3	1,100	.57	.65
	R060508	1-3	1,000	.69	.77
	R060509	1-3	1,000	.70	.83
R10	R060603	1-3	1,000	.70	.80
	R060605	1-3	1,100	.71	.78
	R060606	1-4	1,100	.67	.75
	R060607	1-3	1,100	.62	.69
	R060608	1-3	1,000	.74	.83
R11	R0Q4103	1-3	1,100	.80	.89
	R0Q4104	1-3	1,100	.71	.81
	R0Q4106	1-3	1,000	.63	.74
	R0Q4107	1-4	900	.65	.80
R12	R061003	1-3	1,100	.74	.85
	R061004	1-2	1,100	.78	†
	R061006	1-2	1,000	.81	†
	R061007	1-4	900	.54	.68
	R061010	1-3	900	.68	.78
R13	R069705	1-2	1,100	.73	†
	R069706	1-3	1,100	.73	.83
	R069707	1-4	1,000	.62	.81
	R069708	1-3	1,000	.57	.68
† Not applicable. The intraclass correlation is not reported for dichotomously scored items. NOTE: The intraclass correlation is most appropriate for items with more than two categories. Cohen's Kappa is a measure of reliability that is appropriate for both dichotomously and polytomously scored items. Special codes assigned to student responses including omitted, off-task, and illegible were not included in the calculation of Cohen's Kappa or intraclass correlation. SOURCE: U.S. Department of Education, Institute of Education Sciences, National Center for Education Statistics, National Assessment of Educational Progress (NAEP), 2017 Reading Assessment.

Last updated 11 May 2022 (SK)

Printer-friendly Version