Table of Contents | Search Technical Documentation | References

NAEP Analysis and Scaling → Initial Activities → Constructed-Response Interrater Reliability → Score Range and Percentage of Exact Agreement for the Constructed-Response Items Used in Scaling, Grade 12 Reading National Assessment, by Item and Block: 2015

NAEP Technical DocumentationScore Range and Percentage of Exact Agreement for the Constructed-Response Items Used in Scaling, Grade 12 Reading National Assessment, by Item and Block: 2015

Score range and percentage of exact agreement for the constructed-response items used in scaling, grade 12 reading national assessment, by item and block: 2015
Block	Item	Score range	Sample size	Percentage of exact agreement
R1	R059604	1-2	600	92
	R059605	1-3	600	74
	R059606	1-3	600	86
	R059607	1-3	600	90
	R059610	1-4	600	81
R2	R069801	1-3	600	93
	R069804	1-3	600	81
	R069806	1-4	600	75
	R069810	1-3	600	83
R3	R060103	1-2	600	87
	R060105	1-2	600	95
	R060108	1-4	600	72
R4	R080005	1-4	600	82
	R080007	1-2	600	88
	R080009	1-3	600	81
	R080010	1-3	600	80
R5	R061602	1-3	600	87
	R061604	1-3	600	90
	R061606	1-3	600	80
	R061607	1-4	600	70
	R061608	1-2	600	92
R6	R078903	1-3	600	90
	R078905	1-4	600	74
	R078907	1-2	600	91
R7	R062203	1-3	600	80
	R062205	1-2	600	93
	R062207	1-4	600	76
	R062210	1-3	600	81
R8	R062303	1-3	600	81
	R062304	1-3	600	86
	R062308	1-4	600	74
	R062311	1-3	600	77
R9	R062501	1-3	600	84
	R062503	1-3	600	83
	R062505	1-3	600	80
	R062508	1-4	600	73
R10	R062705	1-3	600	78
	R062706	1-4	600	72
	R062708	1-2	600	93
	R062710	1-3	600	83
R11	R062801	1-3	600	78
	R062805	1-2	600	88
	R062807	1-3	600	77
	R062808	1-4	600	76
R12	R063101	1-3	600	86
	R063105	1-4	600	88
	R063109	1-3	600	98
	R063801	1-3	600	83
R13	R079101	1-3	600	78
	R079106	1-4	600	75
	R079107	1-2	600	91
	R079110	1-3	600	88
	R079111	1-3	600	86
NOTE: Special codes assigned to student responses including blank, off-task, and not-scorable were included in the calculation of the percentage of exact agreement measure. SOURCE: U.S. Department of Education, Institute of Education Sciences, National Center for Education Statistics, National Assessment of Educational Progress (NAEP), 2015 Reading Assessment.

Score range, Cohen's Kappa, and intraclass correlation for the constructed-response items used in scaling, grade 12 reading national assessment, by item and block: 2015
Block	Item	Score range	Sample size	Cohen’s Kappa	Intraclass correlation
R1	R059604	1-2	600	.62	†
	R059605	1-3	500	.55	.72
	R059606	1-3	600	.73	.83
	R059607	1-3	500	.79	.88
	R059610	1-4	500	.71	.88
R2	R069801	1-3	600	.86	.88
	R069804	1-3	600	.68	.79
	R069806	1-4	600	.57	.73
	R069810	1-3	600	.73	.83
R3	R060103	1-2	600	.60	†
	R060105	1-2	600	.85	†
	R060108	1-4	600	.59	.76
R4	R080005	1-4	600	.73	.88
	R080007	1-2	600	.74	†
	R080009	1-3	500	.65	.78
	R080010	1-3	500	.63	.73
R5	R061602	1-3	600	.78	.89
	R061604	1-3	600	.82	.86
	R061606	1-3	500	.65	.80
	R061607	1-4	600	.54	.75
	R061608	1-2	500	.71	†
R6	R078903	1-3	600	.83	.91
	R078905	1-4	600	.57	.75
	R078907	1-2	600	.80	†
R7	R062203	1-3	600	.59	.62
	R062205	1-2	600	.84	†
	R062207	1-4	600	.59	.78
	R062210	1-3	500	.62	.69
R8	R062303	1-3	600	.62	.69
	R062304	1-3	600	.73	.80
	R062308	1-4	500	.57	.79
	R062311	1-3	500	.55	.73
R9	R062501	1-3	600	.68	.80
	R062503	1-3	500	.71	.83
	R062505	1-3	600	.66	.74
	R062508	1-4	500	.58	.79
R10	R062705	1-3	600	.59	.71
	R062706	1-4	600	.59	.80
	R062708	1-2	600	.79	†
	R062710	1-3	600	.72	.80
R11	R062801	1-3	600	.66	.78
	R062805	1-2	600	.73	†
	R062807	1-3	600	.63	.77
	R062808	1-4	500	.60	.80
R12	R063101	1-3	600	.79	.88
	R063105	1-4	500	.77	.89
	R063109	1-3	500	.95	.99
	R063801	1-3	600	.72	.82
R13	R079101	1-3	600	.63	.78
	R079106	1-4	500	.60	.81
	R079107	1-2	600	.80	†
	R079110	1-3	600	.77	.86
	R079111	1-3	500	.75	.81
† Not applicable. The intraclass correlation is not reported for dichotomously scored items. NOTE: The intraclass correlation is most appropriate for items with more than two categories. Cohen's Kappa is a measure of reliability that is appropriate for both dichotomously and polytomously scored items. Special codes assigned to student responses including blank, off-task, and not-scorable were not included in the calculation of Cohen's Kappa or intraclass correlation. SOURCE: U.S. Department of Education, Institute of Education Sciences, National Center for Education Statistics, National Assessment of Educational Progress (NAEP), 2015 Reading Assessment.

Last updated 09 November 2021 (ML)

Printer-friendly Version