NAEP Analysis and Scaling → Initial Activities → Constructed-Response Interrater Reliability → Range of Response Codes, Percent Agreement, and Cohen's Kappa or Intraclass Correlation for the Constructed-Response Items Used in Scaling, Grade 12 Civics National Assessment, by Item and Block: 2010

NAEP Technical DocumentationRange of response codes, percent agreement, and Cohen's Kappa or intraclass correlation for the constructed-response items used in scaling, grade 12 civics national assessment, by item and block: 2010

Block	Item	Range of response codes	Sample size	Percent exact agreement	Cohen’s Kappa	Intraclass correlation
† The intraclass correlation is not reported for dichotomously scored items; Cohen's Kappa is not reported for polytomously scored items. NOTE: Cohen's Kappa is a measure of reliability that is appropriate for items that are dichotomously scored. The intraclass correlation coefficient is most appropriate for items with more than two categories. SOURCE: U.S. Department of Education, Institute of Education Sciences, National Center for Education Statistics, National Assessment of Educational Progress (NAEP), 2010 Civics Assessment.
C3	P041503	1-3	500	80	†	0.77
	P041505	1-3	600	84	†	0.81
	P041509	1-4	500	84	†	0.87
	P041511	1-3	600	86	†	0.89
C4	P041606	1-3	600	78	†	0.78
	P041613	1-3	600	98	†	0.98
	P041614	1-3	600	92	†	0.90
C5	P055401	1-3	600	88	†	0.88
	P056501	1-3	500	86	†	0.83
	P059701	1-4	500	88	†	0.94
	P060801	1-2	600	94	0.84	†
	P062301	1-3	600	89	†	0.84
C6	P041804	1-3	600	71	†	0.73
	P041806	1-3	500	83	†	0.85
	P041815	1-3	600	89	†	0.87
C7	P057201	1-3	600	94	†	0.93
	P058001	1-3	500	92	†	0.93
	P058002	1-3	500	90	†	0.90
	P061201	1-4	600	85	†	0.89
C8	P042191	1-3	600	85	†	0.87
	P042195	1-4	500	77	†	0.85
	P042199	1-3	600	80	†	0.65
	P042203	1-3	500	75	†	0.74
C9	P042210	1-3	500	90	†	0.92
	P042214	1-4	500	67	†	0.79
	P042218	1-4	500	95	†	0.97
	P042222	1-3	500	81	†	0.81
C10	P041404	1-4	600	73	†	0.70
	P041408	1-3	600	87	†	0.89
	P041412	1-3	500	89	†	0.86
	P041413	1-3	500	84	†	0.78

Printer-friendly Version