NAEP Analysis and Scaling → Initial Activities → Constructed-Response Interrater Reliability → Range of Response Codes, Percent Agreement, and Cohen's Kappa or Intraclass Correlation for the Constructed-Response Items Used in Scaling, Grade 8 Civics National Assessment, by Item and Block: 2010

NAEP Technical DocumentationRange of response codes, percent agreement, and Cohen's Kappa or intraclass correlation for the constructed-response items used in scaling, grade 8 civics national assessment, by item and block: 2010

Block	Item	Range of response codes	Sample size	Percent exact agreement	Cohen’s Kappa	Intraclass correlation
† The intraclass correlation is not reported for dichotomously scored items; Cohen's Kappa is not reported for polytomously scored items. NOTE: Cohen's Kappa is a measure of reliability that is appropriate for items that are dichotomously scored. The intraclass correlation coefficient is most appropriate for items with more than two categories. SOURCE: U.S. Department of Education, Institute of Education Sciences, National Center for Education Statistics, National Assessment of Educational Progress (NAEP), 2010 Civics Assessment.
C3	P042171	1-3	600	93	†	0.94
	P042174	1-3	500	90	†	0.71
	P042177	1-3	600	91	†	0.91
	P042181	1-3	600	91	†	0.89
	P042184	1-3	600	86	†	0.83
C4	P051401	1-3	600	89	†	0.87
	P052101	1-4	600	82	†	0.81
	P053801	1-3	600	89	†	0.82
	P054101	1-3	500	83	†	0.76
C5	P040903	1-3	600	70	†	0.64
	P040906	1-3	500	93	†	0.90
	P040910	1-3	600	85	†	0.80
	P040913	1-3	500	95	†	0.94
C6	P049201	1-3	600	89	†	0.90
	P049501	1-3	500	83	†	0.84
	P050501	1-4	600	85	†	0.91
	P054801	1-3	400	96	†	0.97
	P062201	1-3	500	89	†	0.83
C7	P041102	1-3	500	93	†	0.92
	P041106	1-4	600	79	†	0.74
	P041111	1-3	500	85	†	0.79
	P041116	1-3	500	73	†	0.61
C8	P041202	1-3	600	79	†	0.78
	P041205	1-3	600	80	†	0.68
	P041213	1-4	500	81	†	0.83
C9	P041307	1-4	500	84	†	0.89
	P041309	1-3	500	91	†	0.92
	P041315	1-3	500	83	†	0.81
C10	P042152	1-3	600	96	†	0.97
	P042155	1-4	600	81	†	0.90
	P042158	1-3	600	93	†	0.92
	P042162	1-3	500	88	†	0.87

Printer-friendly Version