Table of Contents  |  Search Technical Documentation  |  References

NAEP Analysis and Scaling → Initial Activities → Constructed-Response Interrater Reliability → Range of Response Codes, Percent Agreement, and Cohen's Kappa or Intraclass Correlation for the Constructed-Response Items From Previous Years Rescored in 2006, Grade 8 U.S. History Assessment, by Item and Block: 2006

NAEP Technical DocumentationRange of response codes, percent agreement, and Cohen's Kappa or intraclass correlation for the constructed-response items from previous years rescored in 2006, grade 8 U.S. history assessment, by item and block: 2006

Block	Item	Range of response codes	Sample size	Percent exact agreement	Cohen's Kappa	Intraclass correlation
H3	H035801	1–3	700	84.38	†	0.62
	H035901	1–3	800	90.33	†	0.88
	H035902	1–3	700	79.65	†	0.72
	H036101	1–3	700	87.08	†	0.87
	H036402	1–4	700	73.47	†	0.84
H4	H059001	1–3	800	85.96	†	0.88
	H059201	1–3	700	73.48	†	0.76
	H059701	1–3	700	86.67	†	0.89
	H059801	1–3	700	84.92	†	0.87
	H060201	1–3	700	80.63	†	0.80
H5	H038103	1–4	700	82.82	†	0.60
	H038301	1–3	700	85.73	†	0.88
	H038601	1–3	800	82.60	†	0.66
	H038702	1–3	700	85.71	†	0.79
	H039001	1–3	800	95.08	†	0.94
H7	H057501	1–3	800	94.64	†	0.92
	H057701	1–3	800	82.68	†	0.76
	H057801	1–3	700	77.91	†	0.81
	H058601	1–3	800	80.29	†	0.80
	H058701	1–3	700	86.25	†	0.85
H8	H034101	1–4	800	85.19	†	0.71
	H034401	1–3	800	89.58	†	0.84
	H034501	1–2	800	99.76	0.99	†
	H034702	1–3	800	90.05	†	0.82
	H035001	1–3	800	75.78	†	0.78
	H035101	1–3	800	76.85	†	0.71
H9	H061501	1–3	800	88.37	†	0.83
	H061601	1–3	800	90.00	†	0.86
	H061801	1–3	800	96.00	†	0.93
H10	H042201	1–3	700	94.46	†	0.83
	H042801	1–4	800	88.65	†	0.83
	H042902	1–3	700	88.20	†	0.90
	H043001	1–3	800	78.50	†	0.66
	H043101	1–3	700	91.85	†	0.90
† The intraclass correlation is not reported for dichotomously scored items; Cohen's Kappa is not reported for polytomously scored items. NOTE: Cohen's Kappa is a measure of reliability that is appropriate for items that are dichotomously scored. The intraclass correlation coefficient is most appropriate for items with more than two categories. SOURCE: U.S. Department of Education, Institute of Education Sciences, National Center for Education Statistics, National Assessment of Educational Progress (NAEP), 2006 U.S. History Assessment.

Printer-friendly Version