NAEP Analysis and Scaling → Initial Activities → Constructed-Response Interrater Reliability → Range of Response Codes, Percent Agreement, and Cohen's Kappa or Intraclass Correlation for the Constructed-Response Items Used in Scaling, Grade 12 Reading National Main Assessment, by Item and Block: 2009

NAEP Technical DocumentationRange of response codes, percent agreement, and Cohen's Kappa or intraclass correlation for the constructed-response items used in scaling, grade 12 reading national main assessment, by item and block: 2009

Block	Item	Range of response codes	Sample size	Percent agreement	Cohen’s Kappa	Intraclass correlation
— Not available. The intraclass correlation is not reported for dichotomously scored items; Cohen's Kappa is not reported for polytomously scored items. NOTE: Cohen's Kappa is a measure of reliability that is appropriate for items that are dichotomously scored. The Intraclass Correlation Coefficient is most appropriate for items with more than two categories. SOURCE: U.S. Department of Education, Institute of Education Sciences, National Center for Education Statistics, National Assessment of Educational Progress (NAEP), 2009 Reading Assessment.
F3	R017101	1–2	600	96	0.89	—
	R017102	1–3	600	82	—	0.83
	R017104	1–3	600	92	—	0.93
	R017105	1–4	600	71	—	0.81
	R017107	1–3	600	88	—	0.89
	R017108	1–2	600	95	0.89	—
	R017110	1–3	600	94	—	0.95
F4	R013501	1–2	600	92	0.84	—
	R013503	1–2	600	97	0.93	—
	R013505	1–2	600	94	0.86	—
	R013506	1–4	400	71	—	0.73
	R013508	1–2	600	88	0.75	—
	R013509	1–2	400	89	0.76	—
F5	R016301	1–3	600	86	—	0.82
	R016302	1–3	600	80	—	0.79
	R016303	1–3	600	90	—	0.91
	R016305	1–3	600	78	—	0.75
	R016306	1–3	600	89	—	0.91
	R016307	1–3	600	85	—	0.78
	R016308	1–4	400	71	—	0.79
F6	R013201	1–4	600	85	—	0.92
	R013203	1–2	600	100	0.95	—
	R013205	1–2	600	98	0.87	—
	R013207	1–2	600	94	0.79	—
	R013209	1–2	600	93	0.85	—
	R013211	1–2	600	90	0.79	—
	R013212	1–4	600	86	—	0.83
F7	R013701	1–2	600	91	0.82	—
	R013702	1–2	600	88	0.76	—
	R013704	1–2	600	94	0.79	—
	R013706	1–4	600	73	—	0.72
	R013708	1–2	600	95	0.89	—
	R013710	1–2	600	88	0.69	—
	R013712	1–2	600	81	0.61	—
F8	R053501	1–3	600	87	—	0.83
	R053504	1–2	600	97	0.90	—
	R053507	1–4	600	92	—	0.93
	R053509	1–2	600	90	0.71	—
F9	R053301	1–3	600	94	—	0.88
	R053303	1–2	600	93	0.83	—
	R053305	1–3	600	84	—	0.83
	R053309	1–3	600	87	—	0.87
F10	R013402	1–2	600	99	0.97	—
	R013403	1–4	600	97	—	0.98
	R013405	1–2	600	97	0.91	—
	R013406	1–4	600	86	—	0.94
	R013407	1–2	600	95	0.87	—
	R013409	1–2	600	95	0.85	—
	R013411	1–2	600	91	0.71	—
	R013413	1–2	600	84	0.67	—
F11	R019301	1–4	600	91	—	0.94
	R019401	1–3	600	94	—	0.94
	R019501	1–2	600	91	0.82	—
	R019701	1–3	600	91	—	0.91
	R020001	1–3	600	89	—	0.88
	R020201	1–3	600	90	—	0.89
R1	R059604	1–2	1,500	96	0.77	—
	R059605	1–3	1,400	87	—	0.85
	R059606	1–3	1,400	90	—	0.87
	R059607	1–3	1,300	94	—	0.93
	R059610	1–4	1,300	89	—	0.93
R2	R059803	1–4	1,400	77	—	0.84
	R059806	1–3	1,400	83	—	0.84
	R059807	1–3	1,400	88	—	0.82
	R059809	1–2	1,400	93	0.84	—
R3	R060103	1–2	1,400	88	0.69	—
	R060105	1–2	1,400	93	0.83	—
	R060108	1–4	1,300	76	—	0.77
R4	R060301	1–2	1,500	92	0.73	—
	R060305	1–3	1,400	85	—	0.69
	R060306	1–4	1,400	78	—	0.81
	R060309	1–3	1,400	76	—	0.76
R5	R061602	1–3	1,400	93	—	0.94
	R061604	1–3	1,400	87	—	0.82
	R061606	1–3	1,300	82	—	0.86
	R061607	1–4	1,400	79	—	0.86
	R061608	1–2	1,300	94	0.77	—
R6	R061904	1–2	1,400	97	0.92	—
	R061905	1–2	1,400	88	0.76	—
	R061906	1–3	1,400	84	—	0.83
	R061908	1–3	1,300	82	—	0.82
	R061909	1–4	1,200	79	—	0.81
	R061910	1–3	1,200	83	—	0.78
R7	R062203	1–3	1,400	89	—	0.79
	R062205	1–2	1,400	96	0.93	—
	R062207	1–4	1,300	78	—	0.82
	R062210	1–3	1,300	84	—	0.71
R8	R062303	1–3	1,400	79	—	0.70
	R062304	1–3	1,400	85	—	0.82
	R062308	1–4	1,300	77	—	0.86
	R062311	1–3	1,300	80	—	0.80
R9	R062501	1–3	1,400	91	—	0.89
	R062503	1–3	1,300	85	—	0.87
	R062505	1–3	1,400	81	—	0.78
	R062508	1–4	1,300	67	—	0.76
R10	R062705	1–3	1,400	83	—	0.81
	R062706	1–4	1,400	81	—	0.89
	R062708	1–2	1,400	96	0.88	—
	R062710	1–3	1,300	85	—	0.81
R11	R062902	1–3	1,400	81	—	0.77
	R062903	1–3	1,400	84	—	0.84
	R062908	1–4	1,200	73	—	0.83
	R062909	1–3	1,100	70	—	0.69
R12	R063101	1–3	1,400	89	—	0.90
	R063105	1–4	1,200	91	—	0.93
	R063109	1–3	1,300	99	—	0.99
	R063801	1–3	1,400	91	—	0.91
R13	R063204	1–3	1,400	94	—	0.94
	R063207	1–4	1,300	89	—	0.93
	R063208	1–3	1,300	87	—	0.87
	R063210	1–3	1,300	85	—	0.75

Printer-friendly Version