Table of Contents  |  Search Technical Documentation  |  References

NAEP Analysis and Scaling → Initial Activities → Constructed-Response Interrater Reliability → Range of Response Codes, Percentage Exact Agreement, and Cohen's Kappa or Intraclass Correlation for the Constructed-Response Items Used in Scaling, by Block and Item, Grade 4 U.S. History Assessment: 2001

NAEP Technical DocumentationRange of response codes, percentage exact agreement, and Cohen's Kappa or intraclass correlation for the constructed-response items used in scaling, by block and item, grade 4 U.S. history assessment: 2001

Block	Item	Range of response codes	Sample size	Percentage exact agreement	Cohen's Kappa	Intraclass correlation
H3	H028201	1–4	700	91	†	0.94
	H028701	1–3	700	91	†	0.94
	H028702	1–3	700	91	†	0.95
	H028801	1–3	700	95	†	0.96
	H029002	1–3	700	92	†	0.96
H4	H054301	1–3	700	84	†	0.90
	H054601	1–3	700	92	†	0.94
	H054801	1–3	700	94	†	0.96
	H054901	1–3	700	97	†	0.98
	H055301	1–3	700	88	†	0.93
H5	H055901	1–3	700	95	†	0.97
	H056101	1–3	700	95	†	0.97
	H056401	1–3	700	94	†	0.97
	H056601	1–3	700	96	†	0.97
	H056801	1–3	700	98	†	0.98
H6	H031701	1–4	700	89	†	0.94
	H031801	1–3	700	89	†	0.90
	H031802	1–3	700	95	†	0.98
	H032301	1–3	700	87	†	0.92
	H032503	1–3	700	92	†	0.94
H7	H057501	1–3	600	98	†	0.98
	H057701	1–3	600	90	†	0.92
	H057801	1–3	600	91	†	0.92
	H058601	1–3	600	91	†	0.96
	H058701	1–3	600	94	†	0.96
H8	H034101	1–4	700	96	†	0.95
	H034401	1–3	700	93	†	0.96
	H034501	1–2	700	100	1.00	†
	H034702	1–3	700	99	†	0.98
	H035001	1–3	700	95	†	0.97
	H035101	1–3	700	91	†	0.96
† The intraclass correlation is not reported for dichotomously scored items; Cohen's Kappa is not reported for polytomously scored items. NOTE: Cohen's Kappa is a measure of reliability that is appropriate for items that are dichotomously scored. The intraclass correlation coefficient is most appropriate for items with more than two categories. SOURCE: U.S. Department of Education, Institute of Education Sciences, National Center for Education Statistics, National Assessment of Educational Progress (NAEP), 2001 U.S. History Assessment.

Printer-friendly Version