NAEP Analysis and Scaling → Initial Activities → Constructed-Response Interrater Reliability → Range of Response Codes, Percent Agreement, and Cohen's Kappa or Intraclass Correlation for the Constructed-Response Items Used in Scaling, Grade 12 Economics National Assessment, By Item and Block: 2012

NAEP Technical DocumentationScore range and percent agreement for the constructed-response items used in scaling, grade 12 economics national assessment, by item and block: 2012

Block	Item	Range of response codes	Sample size	Percent exact agreement
NOTE: Special codes assigned to student responses including blank, off-task, and not-scorable were included in the calculation of the percent exact agreement measure. SOURCE: U.S. Department of Education, Institute of Education Sciences, National Center for Education Statistics, National Assessment of Educational Progress (NAEP), 2012 Economics Assessment.
E1	E427501	1-3	600	95
	E419001	1-3	600	93
	E421001	1-3	500	97
E2	E421701	1-3	600	79
	E420301	1-3	600	83
	E422901	1-3	600	86
E3	E403901	1-3	500	97
	E404201	1-3	600	91
	E404601	1-3	500	88
	E405101	1-2	500	91
E4	E420601	1-3	600	97
	E423401	1-3	600	86
E5	E407501	1-4	600	91
	E408001	1-3	500	71
	E408401	1-3	600	92
E6	E409701	1-3	500	94
	E410201	1-5	500	85
	E410601	1-3	500	94
E7	E411401	1-3	600	82
	E411901	1-4	600	89
	E412301	1-3	600	92
E8	E413201	1-4	500	77
	E413701	1-3	600	97
	E414101	1-2	600	92
E9	E415201	1-3	600	71
	E415701	1-2	600	91
	E416101	1-3	600	93
E10	E417201	1-3	600	92
	E417701	1-5	600	74
	E418101	1-3	600	82

Score range and Cohen's Kappa or intraclass correlation for the constructed-response items used in scaling, grade 12 economics national assessment, by item and block: 2012

Block	Item	Range of response codes	Sample size	Cohen’s Kappa	Intraclass correlation
† Not applicable. The intraclass correlation is not reported for dichotomously scored items; Cohen’s Kappa is not reported for polytomously scored items. NOTE:  Cohen's Kappa is a measure of reliability that is used for items that are dichotomously scored. The intraclass correlation is used for items with more than two categories. SOURCE: U.S. Department of Education, Institute of Education Sciences, National Center for Education Statistics, National Assessment of Educational Progress (NAEP), 2012 Economics Assessment.
E1	E427501	1-3	500	†	0.96
	E419001	1-3	500	†	0.96
	E421001	1-3	500	†	0.97
E2	E421701	1-3	500	†	0.80
	E420301	1-3	500	†	0.77
	E422901	1-3	500	†	0.73
E3	E403901	1-3	500	†	0.97
	E404201	1-3	500	†	0.83
	E404601	1-3	500	†	0.86
	E405101	1-2	500	0.73	†
E4	E420601	1-3	500	†	0.98
	E423401	1-3	500	†	0.70
E5	E407501	1-4	500	†	0.92
	E408001	1-3	500	†	0.63
	E408401	1-3	500	†	0.90
E6	E409701	1-3	500	†	0.92
	E410201	1-5	500	†	0.86
	E410601	1-3	500	†	0.93
E7	E411401	1-3	500	†	0.76
	E411901	1-4	500	†	0.84
	E412301	1-3	500	†	0.91
E8	E413201	1-4	500	†	0.82
	E413701	1-3	400	†	0.91
	E414101	1-2	500	0.73	†
E9	E415201	1-3	500	†	0.70
	E415701	1-2	500	0.78	†
	E416101	1-3	500	†	0.92
E10	E417201	1-3	500	†	0.84
	E417701	1-5	500	†	0.86
	E418101	1-3	500	†	0.77

Printer-friendly Version