Table of Contents  |  Search Technical Documentation  |  References

NAEP Analysis and Scaling → Initial Activities → Constructed-Response Interrater Reliability → Range of Response Codes, Percentage Exact Agreement, and Cohen's Kappa or Intraclass Correlation for the Constructed-Response Items From 1994 That Were Rescored in 2001, by Block and Item, Grade 8 U.S. History Assessment: 2001

NAEP Technical DocumentationRange of response codes, percentage exact agreement, and Cohen's Kappa or intraclass correlation for the constructed-response items from 1994 that were rescored in 2001, by block and item, grade 8 U.S. history assessment: 2001

Block	Item	Range of response codes	Sample size	Percentage exact agreement	Cohen's Kappa	Intraclass correlation
H3	H035801	1–3	600	92	†	0.77
	H035901	1–3	600	94	†	0.95
	H035902	1–3	600	84	†	0.77
	H036101	1–3	600	86	†	0.87
	H036402	1–4	600	66	†	0.73
H5	H038103	1–4	600	88	†	0.74
	H038301	1–3	600	91	†	0.92
	H038601	1–3	600	87	†	0.65
	H038702	1–3	600	89	†	0.82
	H039001	1–3	600	95	†	0.95
H6	H039401	1–3	600	89	†	0.83
	H039901	1–4	600	89	†	0.90
	H040001	1–3	600	93	†	0.91
	H040103	1–3	600	81	†	0.78
	H040201	1–3	600	91	†	0.91
H8	H034101	1–4	600	87	†	0.81
	H034401	1–3	600	93	†	0.88
	H034501	1–2	600	100	1.00	†
	H034702	1–3	600	91	†	0.83
	H035001	1–3	600	80	†	0.84
	H035101	1–3	600	88	†	0.78
H10	H042201	1–3	600	99	†	0.94
	H042801	1–4	600	89	†	0.87
	H042902	1–3	600	94	†	0.95
	H043001	1–3	600	85	†	0.77
	H043101	1–3	600	94	†	0.93
H11	H043201	1–3	600	87	†	0.85
	H043401	1–3	600	89	†	0.92
	H043501	1–4	600	81	†	0.88
	H043601	1–3	600	86	†	0.85
	H043701	1–3	600	90	†	0.92
	H043705	1–3	600	78	†	0.76
	H044001	1–4	600	79	†	0.78
† The intraclass correlation is not reported for dichotomously scored items; Cohen's Kappa is not reported for polytomously scored items. NOTE: Cohen's Kappa is a measure of reliability that is appropriate for items that are dichotomously scored. The intraclass correlation coefficient is most appropriate for items with more than two categories. SOURCE: U.S. Department of Education, Institute of Education Sciences, National Center for Education Statistics, National Assessment of Educational Progress (NAEP), 2001 U.S. History Assessment.

Printer-friendly Version