NAEP Analysis and Scaling → Initial Activities → Constructed-Response Interrater Reliability → Range of Response Codes, Percent Agreement, and Cohen's Kappa or Intraclass Correlation for the Constructed-Response Items From Previous Years, Rescored in 2009, Grade 12 Reading National Main Assessment, by Item and Block: 2009

NAEP Technical DocumentationRange of response codes, percent agreement, and Cohen's Kappa or intraclass correlation for the constructed-response items from previous years, rescored in 2009, grade 12 reading national main assessment, by item and block: 2009

Block	Item	Range of response codes	Sample size	Percent agreement	Cohen’s Kappa	Intraclass correlation
— Not available. The intraclass correlation is not reported for dichotomously scored items; Cohen's Kappa is not reported for polytomously scored items. NOTE: Cohen's Kappa is a measure of reliability that is appropriate for items that are dichotomously scored. The Intraclass Correlation Coefficient is most appropriate for items with more than two categories. SOURCE: U.S. Department of Education, Institute of Education Sciences, National Center for Education Statistics, National Assessment of Educational Progress (NAEP), 2009 Reading Assessment.
F3	R017101	1–2	600	91	0.75	—
	R017102	1–3	600	84	—	0.85
	R017104	1–3	600	83	—	0.83
	R017105	1–4	600	66	—	0.71
	R017107	1–3	600	85	—	0.85
	R017108	1–2	600	93	0.86	—
	R017110	1–3	600	91	—	0.92
F4	R013501	1–2	600	90	0.78	—
	R013503	1–2	600	90	0.80	—
	R013505	1–2	600	88	0.73	—
	R013506	1–4	600	68	—	0.73
	R013508	1–2	600	88	0.75	—
	R013509	1–2	600	91	0.79	—
F5	R016301	1–3	600	81	—	0.78
	R016302	1–3	600	78	—	0.74
	R016303	1–3	600	85	—	0.81
	R016305	1–3	600	76	—	0.75
	R016306	1–3	600	82	—	0.85
	R016307	1–3	500	77	—	0.72
	R016308	1–4	500	78	—	0.82
F6	R013201	1–4	600	77	—	0.86
	R013203	1–2	600	97	0.64	—
	R013205	1–2	600	97	0.65	—
	R013207	1–2	600	88	0.64	—
	R013209	1–2	600	93	0.86	—
	R013211	1–2	600	88	0.76	—
	R013212	1–4	600	72	—	0.66
F7	R013701	1–2	600	86	0.71	—
	R013702	1–2	600	88	0.74	—
	R013704	1–2	600	90	0.66	—
	R013706	1–4	500	73	—	0.68
	R013708	1–2	500	86	0.72	—
	R013710	1–2	500	87	0.70	—
	R013712	1–2	500	76	0.53	—
F8	R053501	1–3	600	68	—	0.58
	R053504	1–2	700	96	0.84	—
	R053507	1–4	600	85	—	0.88
	R053509	1–2	600	92	0.74	—
F9	R053301	1–3	600	93	—	0.83
	R053303	1–2	600	88	0.70	—
	R053305	1–3	600	85	—	0.80
	R053309	1–3	600	87	—	0.90
F10	R013402	1–2	600	98	0.94	—
	R013403	1–4	600	92	—	0.95
	R013405	1–2	600	94	0.81	—
	R013406	1–4	500	76	—	0.90
	R013407	1–2	600	87	0.70	—
	R013409	1–2	600	95	0.79	—
	R013411	1–2	600	90	0.69	—
	R013413	1–2	500	80	0.57	—
F11	R019301	1–4	600	93	—	0.95
	R019401	1–3	600	91	—	0.92
	R019501	1–2	600	86	0.70	—
	R019701	1–3	600	88	—	0.88
	R020001	1–3	600	82	—	0.77
	R020201	1–3	600	84	—	0.80

Printer-friendly Version