NAEP Analysis and Scaling → Initial Activities → Constructed-Response Interrater Reliability → Range of Response Codes, Percent Agreement, and Cohen's Kappa or Intraclass Correlation for the Constructed-Response Items From Previous Years, Rescored in 2009, Grade 8 Reading Combined National and State Main Assessment, by Item and Block: 2009

NAEP Technical DocumentationRange of response codes, percent agreement, and Cohen's Kappa or intraclass correlation for the constructed-response items from previous years, rescored in 2009, grade 8 reading combined national and state main assessment, by item and block: 2009

Block	Item	Range of response codes	Sample size	Percent agreement	Cohen’s Kappa	Intraclass correlation
— Not available. The intraclass correlation is not reported for dichotomously scored items; Cohen's Kappa is not reported for polytomously scored items. NOTE: Cohen's Kappa is a measure of reliability that is appropriate for items that are dichotomously scored. The Intraclass Correlation Coefficient is most appropriate for items with more than two categories. SOURCE: U.S. Department of Education, Institute of Education Sciences, National Center for Education Statistics, National Assessment of Educational Progress (NAEP), 2009 Reading Assessment.
F3	R017101	1–2	1,200	90	0.80	—
	R017102	1–3	1,200	87	—	0.85
	R017104	1–3	1,200	91	—	0.91
	R017105	1–4	1,200	74	—	0.81
	R017107	1–3	1,200	84	—	0.85
	R017108	1–2	1,200	96	0.85	—
	R017110	1–3	1,200	92	—	0.92
F4	R055501	1–3	1,200	81	—	0.78
	R055801	1–3	1,200	83	—	0.71
	R056101	1–4	1,200	73	—	0.81
	R056301	1–3	1,200	82	—	0.72
F5	R012601	1–2	1,200	90	0.78	—
	R012604	1–2	1,200	92	0.84	—
	R012607	1–4	1,200	74	—	0.78
	R012612	1–2	1,200	85	0.69	—
F6	R013201	1–4	1,200	78	—	0.85
	R013203	1–2	1,100	95	0.74	—
	R013205	1–2	1,200	96	0.82	—
	R013207	1–2	1,200	88	0.70	—
	R013209	1–2	1,100	93	0.82	—
	R013211	1–2	1,100	87	0.74	—
	R013212	1–4	1,100	73	—	0.75
F7	R034601	1–3	1,200	88	—	0.85
	R034901	1–3	1,100	88	—	0.87
	R035001	1–3	1,200	83	—	0.77
	R035201	1–3	1,200	88	—	0.83
	R035301	1–4	1,200	73	—	0.83
F8	R053401	1–3	1,100	81	—	0.83
	R053402	1–2	1,100	92	0.61	—
	R053405	1–4	1,100	76	—	0.83
	R053408	1–3	1,200	81	—	0.83
	R053410	1–2	1,200	83	0.59	—
F9	R056501	1–2	1,200	96	0.90	—
	R056801	1–3	1,200	85	—	0.79
	R057101	1–3	1,200	77	—	0.71
	R057301	1–3	1,200	74	—	0.75
F10	R013402	1–2	1,200	99	0.97	—
	R013403	1–4	1,200	94	—	0.95
	R013405	1–2	1,200	93	0.84	—
	R013406	1–4	1,200	79	—	0.88
	R013407	1–2	1,200	93	0.86	—
	R013409	1–2	1,200	91	0.81	—
	R013411	1–2	1,200	86	0.68	—
	R013413	1–2	1,200	81	0.62	—
F11	R013001	1–2	1,200	88	0.68	—
	R013003	1–2	1,200	99	0.98	—
	R013004	1–4	1,200	71	—	0.84
	R013005	1–2	1,200	90	0.74	—
	R013007	1–2	1,200	96	0.86	—
	R013008	1–2	1,100	82	0.63	—
	R013009	1–2	1,200	89	0.62	—
	R013010	1–2	1,100	94	0.78	—
	R013011	1–2	1,200	86	0.68	—
F12	R053202	1–3	1,100	84	—	0.79
	R053203	1–3	1,200	78	—	0.75
	R053205	1–4	1,100	75	—	0.87
	R053207	1–3	1,100	79	—	0.72
	R053209	1–3	1,100	85	—	0.80
F14	R026501	1–3	1,200	82	—	0.82
	R026801	1–4	1,200	75	—	0.82
	R027001	1–2	1,200	90	0.70	—
	R027201	1–3	1,200	86	—	0.85
F15	R028401	1–3	1,200	90	—	0.89
	R028501	1–3	1,200	89	—	0.87
	R028801	1–4	1,200	77	—	0.86
	R029601	1–3	1,200	80	—	0.76
R1	R059604	1–2	1,200	91	0.67	—
	R059605	1–3	1,100	77	—	0.72
	R059606	1–3	1,100	82	—	0.81
	R059607	1–3	1,100	88	—	0.88
	R059610	1–4	1,100	82	—	0.89
R2	R059803	1–4	1,100	72	—	0.76
	R059806	1–3	1,100	82	—	0.77
	R059807	1–3	1,100	87	—	0.81
	R059809	1–2	1,100	91	0.81	—
R3	R060103	1–2	1,200	83	0.62	—
	R060105	1–2	1,100	92	0.84	—
	R060108	1–4	1,100	69	—	0.74
R4	R060301	1–2	1,200	90	0.72	—
	R060305	1–3	1,200	74	—	0.60
	R060306	1–4	1,100	56	—	0.58
	R060309	1–3	1,100	71	—	0.65
R5	R058504	1–3	1,100	77	—	0.67
	R058508	1–4	1,100	76	—	0.83
	R058509	1–2	1,200	89	0.76	—
R6	R058602	1–3	1,200	80	—	0.70
	R058606	1–3	1,100	80	—	0.75
	R058608	1–4	1,100	73	—	0.80
R7	R058805	1–3	1,200	82	—	0.77
	R058807	1–4	1,100	71	—	0.80
	R058810	1–3	1,100	72	—	0.64
R8	R059106	1–4	1,100	61	—	0.68
	R059108	1–3	1,100	77	—	0.82
	R059109	1–3	1,200	76	—	0.75
	R059110	1–2	1,100	90	0.76	—
R9	R060503	1–3	1,200	93	—	0.93
	R060505	1–4	1,100	81	—	0.88
	R060507	1–3	1,100	79	—	0.64
	R060508	1–3	1,100	74	—	0.61
	R060509	1–3	1,100	82	—	0.81
R10	R060703	1–3	1,200	64	—	0.57
	R060704	1–3	1,100	78	—	0.80
	R060706	1–4	1,100	76	—	0.79
	R060707	1–3	1,100	80	—	0.77
	R060708	1–3	1,100	79	—	0.67
R11	R060803	1–3	1,200	69	—	0.63
	R060807	1–2	1,100	93	0.84	—
	R060808	1–4	1,100	73	—	0.83
	R060810	1–3	1,100	72	—	0.70
	R060811	1–3	1,600	80	—	0.76
R12	R061104	1–3	1,200	78	—	0.75
	R061106	1–4	1,100	64	—	0.66
	R061110	1–2	1,100	86	0.65	—
	R061111	1–3	1,100	87	—	0.82
R13	R061301	1–3	1,200	91	—	0.88
	R061303	1–3	1,200	75	—	0.68
	R061304	1–2	1,100	85	0.68	—
	R061307	1–4	1,100	62	—	0.79
	R061309	1–3	1,100	73	—	0.73

Printer-friendly Version