Table of Contents  |  Search Technical Documentation  |  References

NAEP Analysis and Scaling → Initial Activities → Constructed-Response Interrater Reliability → Range of Response Codes, Percentage Exact Agreement, and Cohen's Kappa or Intraclass Correlation for the Constructed-Response Items Used in Scaling, by Block and Item, Grade 4 Reading Combined National and State Main Assessment: 2002

NAEP Technical DocumentationRange of response codes, percentage exact agreement, and Cohen's Kappa or intraclass correlation for the constructed-response items used in scaling, by block and item, grade 4 reading combined national and state main assessment: 2002

Block	Item	Range of response codes	Sample size	Percent exact agreement	Cohen's Kappa	Intraclass correlation
R3	R017001	1–2	2,800	82	0.62	†
	R017003	1–3	5,100	73	†	0.76
	R017004	1–2	1,800	89	0.72	†
	R017006	1–2	1,200	90	0.72	†
	R017007	1–4	3,900	72	†	0.84
	R017009	1–3	2,600	85	†	0.78
R4	R012102	1–2	2,400	95	0.90	†
	R012104	1–2	1,500	94	0.88	†
	R012106	1–2	2,100	89	0.78	†
	R012108	1–2	900	94	0.87	†
	R012109	1–2	2,800	92	0.83	†
	R012111	1–4	3,400	84	†	0.88
	R012112	1–2	800	88	0.75	†
R5	R012601	1–2	4,500	86	0.69	†
	R012604	1–2	1,800	93	0.83	†
	R012607	1–4	2,700	79	†	0.83
	R012611	1–2	2,500	87	0.73	†
R6	R017301	1–2	1,800	90	0.80	†
	R017303	1–3	1,600	82	†	0.84
	R017310	1–3	4,000	87	†	0.88
	R017307	1–4	4,600	76	†	0.86
	R017309	1–3	3,000	82	†	0.84
R7	R012702	1–2	2,400	89	0.76	†
	R012703	1–2	1,700	87	0.73	†
	R012705	1–2	2,800	89	0.73	†
	R012706	1–2	3,300	82	0.60	†
	R012708	1–4	5,400	82	†	0.82
	R012710	1–2	2,800	88	0.76	†
R9	R015802	1–2	2,600	87	0.74	†
	R015803	1–3	3,900	85	†	0.83
	R015804	1–4	5,100	76	†	0.81
	R015806	1–3	3,200	82	†	0.81
	R015807	1–3	4,200	81	†	0.80
	R015809	1–3	3,100	85	†	0.84
R10	R012503	1–2	1,900	87	0.73	†
	R012504	1–2	1,300	97	0.94	†
	R012506	1–2	2,400	89	0.79	†
	R012508	1–2	1,700	97	0.93	†
	R012511	1–2	2,500	94	0.86	†
	R012512	1–4	3,700	79	†	0.88
† The intraclass correlation is not reported for dichotomously scored items; Cohen's Kappa is not reported for polytomously scored items. NOTE: Cohen's Kappa is a measure of reliability that is appropriate for items that are dichotomously scored. The intraclass correlation coefficient is most appropriate for items with more than two categories. SOURCE: U.S. Department of Education, Institute of Education Sciences, National Center for Education Statistics, National Assessment of Educational Progress (NAEP), 2002 Reading Assessment.

Printer-friendly Version