Table of Contents  |  Search Technical Documentation  |  References

NAEP Analysis and Scaling → Initial Activities → Constructed-Response Interrater Reliability → Range of Response Codes, Percentage Exact Agreement, and Cohen's Kappa or Intraclass Correlation for the Constructed-Response Items From Previous Years, Rescored in 2002, by Block and Item, Grade 12 Reading National Main Assessment: 2002

NAEP Technical DocumentationRange of response codes, percent agreement, and Cohen's Kappa or intraclass correlation for the constructed-response items from previous years, rescored in 2002, by block and item, grade 12 reading national main assessment: 2002

Block	Item	Range of response codes	Sample size	Percent exact agreement	Cohen's Kappa	Intraclass correlation
R3	R017101	1–2	1,000	92	0.84	†
	R017102	1–3	1,400	82	†	0.85
	R017104	1–3	600	88	†	0.90
	R017105	1–4	800	68	†	0.82
	R017107	1–3	600	88	†	0.90
	R017108	1–2	1,400	94	0.88	†
	R017110	1–3	800	91	†	0.93
R4	R013501	1–2	2,200	90	0.80	†
	R013503	1–2	900	94	0.87	†
	R013505	1–2	1,400	87	0.69	†
	R013506	1–4	2,600	76	†	0.75
	R013508	1–2	900	90	0.78	†
	R013509	1–2	1,900	90	0.79	†
R5	R016301	1–3	800	81	†	0.79
	R016302	1–3	1,000	82	†	0.78
	R016303	1–3	900	82	†	0.76
	R016305	1–3	2,400	79	†	0.79
	R016306	1–3	800	85	†	0.87
	R016307	1–3	600	79	†	0.77
	R016308	1–4	800	83	†	0.87
R6	R013201	1–4	2,500	81	†	0.88
	R013203	1–2	1,800	94	0.53	†
	R013205	1–2	600	98	0.80	†
	R013207	1–2	900	90	0.71	†
	R013209	1–2	1,400	96	0.91	†
	R013211	1–2	3,800	81	0.63	†
	R013212	1–4	2,900	81	†	0.77
R7	R013701	1–2	2,000	82	0.64	†
	R013702	1–2	1,400	82	0.65	†
	R013704	1–2	1,400	86	0.56	†
	R013706	1–4	1,000	77	†	0.75
	R013708	1–2	1,400	85	0.70	†
	R013710	1–2	1,900	90	0.71	†
	R013712	1–2	900	81	0.62	†
R8	R016401	1–3	800	82	†	0.77
	R016402	1–3	1,400	62	†	0.61
	R016403	1–3	1,500	78	†	0.76
	R016404	1–3	1,500	86	†	0.71
	R016405	1–3	1,100	88	†	0.80
	R016407	1–3	1,000	80	†	0.83
	R016408	1–4	1,400	82	†	0.84
R9	R016101	1–3	900	90	†	0.87
	R016104	1–3	2,400	82	†	0.58
	R016107	1–3	1,900	93	†	0.90
	R016108	1–3	2,900	75	†	0.74
	R016109	1–3	1,500	89	†	0.86
R10	R013402	1–2	1,100	98	0.96	†
	R013403	1–4	1,500	96	†	0.97
	R013405	1–2	1,400	93	0.81	†
	R013406	1–4	2,000	82	†	0.93
	R013407	1–2	1,700	92	0.79	†
	R013409	1–2	800	89	0.66	†
	R013411	1–2	1,600	94	0.82	†
	R013412	1–2	1,800	86	0.59	†
R13	R015503	1–2	1,600	93	0.57	†
	R015505	1–2	1,400	88	0.76	†
	R015507	1–4	1,000	82	†	0.85
	R015509	1–2	1,500	86	0.70	†
	R015512	1–2	1,100	91	0.80	†
	R015514	1–4	600	85	†	0.85
R14	R016501	1–3	900	82	†	0.70
	R016502	1–3	1,100	84	†	0.77
	R016601	1–3	1,900	78	†	0.72
	R016602	1–3	1,400	85	†	0.84
	R016603	1–3	3,000	70	†	0.58
	R016604	1–3	2,700	79	†	0.75
	R016605	1–3	1,300	79	†	0.69
	R016701	1–4	1,500	76	†	0.70
†The intraclass correlation is not reported for dichotomously scored items; Cohen's Kappa is not reported for polytomously scored items. NOTE: Cohen's Kappa is a measure of reliability that is appropriate for items that are dichotomously scored. The intraclass correlation coefficient is most appropriate for items with more than two categories. SOURCE: U.S. Department of Education, Institute of Education Sciences, National Center for Education Statistics, National Assessment of Educational Progress (NAEP), 2002 Reading Assessment.

Printer-friendly Version