Table of Contents | Search Technical Documentation | References

NAEP Analysis and Scaling → Initial Activities → Constructed-Response Interrater Reliability → Score Range and Percentage of Exact Agreement for the Constructed-Response Items Used in Scaling, Grade 4 Reading Combined National and State Assessment, by Block and Item: 2017

NAEP Technical DocumentationScore Range and Percentage of Exact Agreement for the Constructed-Response Items Used in Scaling, Grade 4 Reading Combined National and State Assessment, by Block and Item: 2017

Score range and percentage of exact agreement for the constructed-response items used in scaling, grade 4 reading combined national and state assessment, by block and item: 2017
Block	Item	Score range	Sample size	Percentage of exact agreement
R1	R068904	1-2	1,400	94
	R068907	1-4	1,400	80
	R068910	1-3	1,300	82
R2	R057806	1-4	1,400	83
	R057807	1-2	1,400	95
	R057809	1-3	1,300	84
R3	R079306	1-3	1,400	83
	R079307	1-4	1,300	79
	R079309	1-3	1,300	83
R4	R070204	1-3	1,400	92
	R070206	1-4	1,400	84
	R070210	1-3	1,300	80
R5	R079504	1-3	1,400	88
	R079507	1-4	1,400	79
	R079508	1-2	1,300	90
	R079510	1-3	1,300	78
R6	R058602	1-3	1,400	81
	R058606	1-3	1,300	89
	R058608	1-4	1,300	84
R7	R058805	1-3	1,400	87
	R058807	1-4	1,400	74
	R058810	1-3	1,300	81
R8	R079703	1-3	1,400	86
	R079706	1-3	1,400	94
	R079707	1-4	1,300	95
	R079709	1-3	1,200	86
R9	R0P4102	1-3	1,400	78
	R0P4108	1-2	1,400	92
	R0P4109	1-4	1,300	81
	R0P4110	1-3	1,300	81
R10	R069305	1-2	1,400	93
R10	R069308	1-4	1,400	76
NOTE: Special codes assigned to student responses including omitted, off-task, and illegible were included in the calculation of the percentage of exact agreement measure. SOURCE: U.S. Department of Education, Institute of Education Sciences, National Center for Education Statistics, National Assessment of Educational Progress (NAEP), 2017 Reading Assessment.

Score range, Cohen's Kappa, and intraclass correlation for the constructed-response items used in scaling, grade 4 reading combined national and state assessment, by block and item: 2017
Block	Item	Score range	Sample size	Cohen’s Kappa	Intraclass correlation
R1	R068904	1-2	1,400	.88	†
	R068907	1-4	1,400	.70	.81
	R068910	1-3	1,300	.72	.82
R2	R057806	1-4	1,400	.74	.86
	R057807	1-2	1,400	.89	†
	R057809	1-3	1,300	.73	.78
R3	R079306	1-3	1,300	.74	.86
	R079307	1-4	1,300	.70	.81
	R079309	1-3	1,300	.71	.81
R4	R070204	1-3	1,400	.84	.87
	R070206	1-4	1,400	.74	.87
	R070210	1-3	1,300	.69	.81
R5	R079504	1-3	1,400	.78	.86
	R079507	1-4	1,400	.70	.87
	R079508	1-2	1,300	.80	†
	R079510	1-3	1,300	.63	.75
R6	R058602	1-3	1,400	.69	.78
	R058606	1-3	1,300	.80	.81
	R058608	1-4	1,300	.74	.84
R7	R058805	1-3	1,400	.80	.86
	R058807	1-4	1,300	.64	.83
	R058810	1-3	1,300	.69	.79
R8	R079703	1-3	1,400	.78	.84
	R079706	1-3	1,400	.89	.95
	R079707	1-4	1,300	.94	.97
	R079709	1-3	1,200	.72	.76
R9	R0P4102	1-3	1,400	.65	.74
	R0P4108	1-2	1,400	.84	†
	R0P4109	1-4	1,300	.73	.87
	R0P4110	1-3	1,300	.67	.75
R10	R069305	1-2	1,400	.82	†
R10	R069308	1-4	1,300	.61	.76
† Not applicable. The intraclass correlation is not reported for dichotomously scored items. NOTE: The intraclass correlation is most appropriate for items with more than two categories. Cohen's Kappa is a measure of reliability that is appropriate for both dichotomously and polytomously scored items. Special codes assigned to student responses including omitted, off-task, and illegible were not included in the calculation of Cohen's Kappa or intraclass correlation. SOURCE: U.S. Department of Education, Institute of Education Sciences, National Center for Education Statistics, National Assessment of Educational Progress (NAEP), 2017 Reading Assessment.

Last updated 11 May 2022 (SK)

Printer-friendly Version