Table of Contents  |  Search Technical Documentation  |  References

NAEP Analysis and Scaling → Initial Activities → Constructed-Response Interrater Reliability → Range of Response Codes, Percentage Exact Agreement, and Cohen's Kappa or Intraclass Correlation for the Constructed-Response Items Used in Scaling, by Block and Item, Grade 8 Reading Combined National and State Main Assessment: 2002

NAEP Technical DocumentationRange of response codes, percentage exact agreement, and Cohen's Kappa or intraclass correlation for the constructed-response items used in scaling, by block and item, grade 8 reading combined national and state main assessment: 2002

Block	Item	Range of response codes	Sample size	Percent exact agreement	Cohen's Kappa	Intraclass correlation
R3	R017101	1–2	600	94	0.87	†
	R017102	1–2	2,200	89	0.83	†
	R017104	1–2	900	94	0.90	†
	R017105	1–2	1,800	74	0.65	†
	R017107	1–2	900	93	0.90	†
	R017108	1–2	1,400	96	0.92	†
	R017110	1–2	1,000	94	0.91	†
R5	R012601	1–2	4,300	84	0.69	†
	R012604	1–2	2,600	91	0.82	†
	R012607	1–2	5,300	75	0.66	†
	R012611	1–2	3,000	87	0.72	†
R6	R013201	1–2	1,600	85	0.78	†
	R013203	1–2	1,200	93	0.69	†
	R013205	1–2	600	94	0.80	†
	R013207	1–2	1,200	93	0.85	†
	R013209	1–2	900	97	0.90	†
	R013211	1–2	1,900	85	0.68	†
	R013212	1–2	2,200	84	0.72	†
R7	R012702	1–2	2,000	93	0.72	†
	R012705	1–2	2,600	87	0.73	†
	R012706	1–2	1,900	83	0.63	†
	R012703	1–2	1,700	84	0.67	†
	R012708	1–2	3,200	79	0.70	†
	R012710	1–2	900	88	0.72	†
	R012713	1–2	1,600	98	0.96	†
R8	R017202	1–2	2,500	85	0.77	†
	R017204	1–2	1,600	81	0.71	†
	R017205	1–2	4,400	57	0.43	†
	R017207	1–2	1,300	88	0.82	†
	R017208	1–2	1,200	81	0.72	†
	R017210	1–2	900	86	0.72	†
R9	R016101	1–2	1,600	86	0.78	†
	R016104	1–2	3,300	83	0.62	†
	R016107	1–2	1,900	93	0.89	†
	R016108	1–2	2,300	76	0.62	†
	R016109	1–2	1,700	88	0.79	†
R10	R013402	1–2	2,300	98	0.96	†
	R013403	1–2	1,800	96	0.93	†
	R013405	1–2	3,400	90	0.78	†
	R013406	1–2	3,100	80	0.73	†
	R013407	1–2	1,700	92	0.83	†
	R013409	1–2	1,600	89	0.76	†
	R013411	1–2	1,800	94	0.85	†
	R013412	1–2	3,500	86	0.60	†
R11	R013001	1–2	1,800	91	0.76	†
	R013003	1–2	1,600	99	0.99	†
	R013004	1–2	2,100	69	0.58	†
	R013005	1–2	1,700	90	0.73	†
	R013007	1–2	1,500	98	0.92	†
	R013008	1–2	1,900	89	0.77	†
	R013009	1–2	2,400	93	0.73	†
	R013010	1–2	1,600	92	0.71	†
	R013011	1–2	2,000	81	0.61	†
R13	R016201	1–2	1,700	92	0.49	†
	R016202	1–2	2,600	86	0.64	†
	R016204	1–2	1,900	88	0.77	†
	R016205	1–2	1,300	87	0.74	†
	R016207	1–2	1,400	89	0.84	†
	R016210	1–2	3,400	70	0.52	†
	R016211	1–2	2,300	88	0.76	†
	R016212	1–2	1,300	91	0.86	†
	R016213	1–2	2,000	88	0.76	†
† The intraclass correlation is not reported for dichotomously scored items; Cohen's Kappa is not reported for polytomously scored items. NOTE: Cohen's Kappa is a measure of reliability that is appropriate for items that are dichotomously scored. The intraclass correlation coefficient is most appropriate for items with more than two categories. SOURCE: U.S. Department of Education, Institute of Education Sciences, National Center for Education Statistics, National Assessment of Educational Progress (NAEP), 2002 Reading Assessment.

Printer-friendly Version