NAEP Analysis and Scaling → Initial Activities → Constructed-Response Interrater Reliability → Range of Response Codes, Percent Agreement, and Cohen's Kappa or Intraclass Correlation for the Constructed-Response Items From Previous Years, Rescored in 2009, Grade 4 Reading Combined National and State Main Assessment, by Item and Block: 2009

NAEP Technical DocumentationRange of response codes, percent agreement, and Cohen's Kappa or intraclass correlation for the constructed-response items from previous years, rescored in 2009, grade 4 reading combined national and state main assessment, by item and block: 2009

Block	Item	Range of response codes	Sample size	Percent agreement	Cohen’s Kappa	Intraclass correlation
— Not available. The intraclass correlation is not reported for dichotomously scored items; Cohen's Kappa is not reported for polytomously scored items. NOTE: Cohen's Kappa is a measure of reliability that is appropriate for items that are dichotomously scored. The Intraclass Correlation Coefficient is most appropriate for items with more than two categories. SOURCE: U.S. Department of Education, Institute of Education Sciences, National Center for Education Statistics, National Assessment of Educational Progress (NAEP), 2009 Reading Assessment.
F3	R053901	1–3	1,200	83	—	0.77
	R054001	1–3	1,200	80	—	0.69
	R054201	1–4	1,200	69	—	0.80
	R054301	1–3	1,200	81	—	0.72
F4	R052904	1–3	1,200	85	—	0.86
	R052906	1–2	1,200	90	0.78	—
	R052907	1–4	1,200	71	—	0.80
	R052909	1–2	1,200	93	0.83	—
F5	R012601	1–2	1,200	89	0.73	—
	R012604	1–2	1,200	94	0.81	—
	R012607	1–4	1,200	83	—	0.88
	R012612	1–2	1,200	89	0.76	—
F6	R017301	1–2	1,200	89	0.71	—
	R017303	1–3	1,200	85	—	0.86
	R017307	1–4	1,200	74	—	0.80
	R017309	1–3	1,200	79	—	0.81
	R017310	1–3	1,200	94	—	0.94
F7	R034601	1–3	1,200	86	—	0.87
	R034901	1–3	1,200	88	—	0.86
	R035001	1–3	1,200	79	—	0.81
	R035201	1–3	1,200	87	—	0.85
	R035301	1–4	1,200	80	—	0.88
F8	R054801	1–3	1,200	96	—	0.97
	R055101	1–4	1,200	85	—	0.94
	R055301	1–3	1,300	79	—	0.78
	R055401	1–3	1,200	90	—	0.89
F9	R053003	1–2	1,200	96	0.91	—
	R053004	1–3	1,200	91	—	0.92
	R053006	1–4	1,200	78	—	0.90
	R053009	1–2	1,300	90	0.72	—
	R053010	1–3	1,200	86	—	0.85
F10	R053101	1–2	1,200	88	0.60	—
	R053105	1–3	1,300	78	—	0.78
	R053106	1–4	1,200	80	—	0.91
	R053108	1–3	1,200	87	—	0.85
F11	R020401	1–3	1,200	92	—	0.91
	R020701	1–2	1,200	90	0.75	—
	R021201	1–3	1,200	82	—	0.78
	R021701	1–3	1,200	80	—	0.74
F12	R023201	1–2	1,200	87	0.73	—
	R023501	1–3	1,200	85	—	0.89
	R023601	1–3	1,200	86	—	0.85
	R024001	1–3	1,200	79	—	0.76
R1	R057704	1–2	1,200	88	0.74	—
	R057705	1–3	1,100	84	—	0.83
	R057706	1–3	1,100	82	—	0.77
	R057707	1–4	1,100	74	—	0.83
	R057710	1–3	1,100	77	—	0.73
R2	R057806	1–4	1,100	79	—	0.82
	R057807	1–2	1,100	95	0.82	—
	R057809	1–3	1,100	82	—	0.75
R3	R058003	1–3	1,200	96	—	0.91
	R058004	1–3	1,100	81	—	0.81
	R058006	1–4	1,100	83	—	0.83
	R058007	1–2	1,100	92	0.83	—
	R058008	1–3	1,100	89	—	0.81
R4	R058204	1–3	1,100	87	—	0.87
	R058206	1–4	1,100	81	—	0.85
	R058207	1–3	1,100	89	—	0.90
	R058209	1–2	1,100	96	0.86	—
R5	R058504	1–3	1,100	75	—	0.66
	R058508	1–4	1,100	73	—	0.84
	R058509	1–2	1,200	93	0.84	—
R6	R058602	1–3	1,100	73	—	0.67
	R058606	1–3	1,100	78	—	0.75
	R058608	1–4	1,100	77	—	0.83
R7	R058805	1–3	1,100	81	—	0.79
	R058807	1–4	1,100	67	—	0.78
	R058810	1–3	1,100	71	—	0.71
R8	R059106	1–4	1,100	61	—	0.66
	R059108	1–3	1,100	82	—	0.83
	R059109	1–3	1,100	76	—	0.70
	R059110	1–2	1,100	93	0.86	—
R9	R059403	1–3	1,100	77	—	0.73
	R059406	1–4	1,100	70	—	0.79
	R059407	1–3	1,100	82	—	0.83
	R059409	1–2	1,100	92	0.83	—
R10	R059504	1–3	1,100	72	—	0.67
	R059507	1–4	1,100	68	—	0.73
	R059509	1–3	1,100	79	—	0.79
	R059510	1–3	1,100	84	—	0.83

Printer-friendly Version