Table of Contents  |  Search Technical Documentation  |  References

NAEP Analysis and Scaling → Initial Activities → Constructed-Response Interrater Reliability → Range of Response Codes, Percent Agreement, and Cohen's Kappa or Intraclass Correlation for the Constructed-Response Items From Previous Years Rescored in 2006, Grade 12 U.S. History Assessment, by Item and Block: 2006

NAEP Technical DocumentationRange of response codes, percent agreement, and Cohen's Kappa or intraclass correlation for the constructed-response items from previous years rescored in 2006, grade 12 U.S. history assessment, by item and block: 2006

Block	Item	Range of response codes	Sample size	Percent exact agreement	Cohen's Kappa	Intraclass correlation
H4	H045901	1–4	700	75.40	†	0.75
	H046001	1–2	800	99.22	0.98	†
	H046101	1–3	700	95.69	†	0.95
	H046301	1–3	700	83.15	†	0.79
H5	H062001	1–3	800	74.00	†	0.68
	H062201	1–3	800	81.28	†	0.77
	H063101	1–3	700	85.25	†	0.83
	H063401	1–4	800	78.00	†	0.86
	H063601	1–3	700	77.04	†	0.74
H6	H048901	1–3	700	92.41	†	0.95
	H049401	1–4	800	85.07	†	0.40
	H049503	1–3	800	76.72	†	0.60
	H049601	1–4	700	81.95	†	0.77
	H049701	1–3	800	94.73	†	0.96
H8	H051301	1–3	800	68.92	†	0.59
	H052301	1–4	800	50.00	†	0.58
	H052501	1–3	700	85.10	†	0.89
	H052701	1–3	800	82.02	†	0.84
H9	H061501	1–3	800	76.79	†	0.79
	H061601	1–3	800	91.50	†	0.88
	H061801	1–3	700	90.07	†	0.86
H10	H042201	1–3	700	92.08	†	0.75
	H042801	1–4	800	85.86	†	0.89
	H042902	1–3	800	89.63	†	0.91
	H043001	1–3	800	72.98	†	0.77
	H043101	1–3	800	88.76	†	0.87
H11	H063801	1–3	800	74.84	†	0.65
	H064001	1–3	800	78.39	†	0.73
	H064101	1–3	800	74.18	†	0.76
	H064401	1–4	700	67.68	†	0.74
	H064901	1–3	800	78.88	†	0.80
	H065101	1–3	800	73.18	†	0.66
	H065201	1–3	800	66.87	†	0.65
	H065301	1–3	700	73.81	†	0.74
	H065401	1–4	700	67.47	†	0.77
† The intraclass correlation is not reported for dichotomously scored items; Cohen's Kappa is not reported for polytomously scored items. NOTE: Cohen's Kappa is a measure of reliability that is appropriate for items that are dichotomously scored. The intraclass correlation coefficient is most appropriate for items with more than two categories. SOURCE: U.S. Department of Education, Institute of Education Sciences, National Center for Education Statistics, National Assessment of Educational Progress (NAEP), 2006 U.S. History Assessment.

Printer-friendly Version