Table of Contents  |  Search Technical Documentation  |  References

NAEP Analysis and Scaling → Initial Activities → Constructed-Response Interrater Reliability → Range of Response Codes, Percentage Exact Agreement, and Cohen's Kappa or Intraclass Correlation for the Constructed-Response Items From 1994 That Were Rescored in 2001, by Block and Item, Grade 4 U.S. History Assessment: 2001

NAEP Technical DocumentationRange of response codes, percentage exact agreement, and Cohen's Kappa or intraclass correlation for the constructed-response items from 1994 that were rescored in 2001, by block and item, grade 4 U.S. history assessment: 2001

Block	Item	Range of response codes	Sample size	Percentage exact agreement	Cohen's Kappa	Intraclass correlation
H3	H028201	1–4	600	88	†	0.90
	H028701	1–3	600	90	†	0.87
	H028702	1–3	600	89	†	0.86
	H028801	1–3	600	88	†	0.86
	H029002	1–3	600	86	†	0.85
H6	H031701	1–4	600	86	†	0.90
	H031801	1–3	600	89	†	0.87
	H031802	1–3	600	96	†	0.97
	H032301	1–3	600	83	†	0.77
	H032503	1–3	600	86	†	0.73
H8	H034101	1–4	600	94	†	0.79
	H034401	1–3	600	95	†	0.94
	H034501	1–2	700	100	0.97	†
	H034702	1–3	600	98	†	0.94
	H035001	1–3	600	89	†	0.87
	H035101	1–3	600	88	†	0.88
† The intraclass correlation is not reported for dichotomously scored items; Cohen's Kappa is not reported for polytomously scored items. NOTE: Cohen's Kappa is a measure of reliability that is appropriate for items that are dichotomously scored. The intraclass correlation coefficient is most appropriate for items with more than two categories. SOURCE: U.S. Department of Education, Institute of Education Sciences, National Center for Education Statistics, National Assessment of Educational Progress (NAEP), 2001 U.S. History Assessment.

Printer-friendly Version