NAEP Analysis and Scaling → Initial Activities → Constructed-Response Interrater Reliability → Range of Response Codes, Percent Agreement, and Cohen's Kappa or Intraclass Correlation for the Constructed-Response Items Used in Scaling, Grade 8 Mathematics Combined National and State Assessments, by Item and Block: 2011

NAEP Technical DocumentationScore range and percent agreement for the constructed-response items used in scaling, grade 8 mathematics combined national and state assessments, by item and block: 2011

Block	Item	Range of response codes	Sample size	Percent exact agreement
NOTE: Special codes assigned to student responses including blank, off-task, and not-scorable were included in the calculation of the Percent Exact Agreement. SOURCE: U.S. Department of Education, Institute of Education Sciences, National Center for Education Statistics, National Assessment of Educational Progress (NAEP), 2011 Mathematics Assessment.
M3	M149901	1-3	1,600	98
	M150801	1-3	1,700	97
	M150901	1-2	1,600	99
	M151201	1-2	1,600	99
M4	M151501	1-3	1,700	94
	M151601	1-3	1,700	95
	M152602	1-5	1,700	97
M6	M141301	1-3	1,600	95
	M141601	1-2	1,600	90
	M141901	1-5	1,600	88
M7	M165301	1-3	1,600	93
	M167001	1-2	1,600	91
	M168501	1-2	1,600	99
	M168502	1-2	1,600	98
	M168503	1-3	1,600	91
	M168701	1-3	1,600	96
	M170101	1-3	1,600	98
M8	M164601	1-3	1,600	98
	M166301	1-3	1,600	95
	M169901	1-2	1,600	99
	M169902	1-2	1,600	100
	M169903	1-3	1,600	93
	M169904	1-2	1,600	94
	M170501	1-3	1,600	90
M9	M165001	1-2	1,600	97
	M165501	1-3	1,600	97
	M165701	1-3	1,600	97
	M165702	1-3	1,600	93
	M168901	1-2	1,600	99
	M170801	1-3	1,600	97
M11	M104901	1-3	1,600	97
	M162901	1-3	1,600	96
	M163101	1-3	1,600	95
	M167901	1-3	1,600	95
	M168301	1-3	1,600	95
M12	M153201	1-3	1,600	94
	M153601	1-2	1,600	99
	M153901	1-3	1,600	92

Score range and Cohen's Kappa or intraclass correlation for the constructed-response items used in scaling, grade 8 mathematics combined national and state assessments, by item and block: 2011

Block	Item	Range of response codes	Sample size	Cohen’s Kappa	Intraclass correlation
† Not applicable. The intraclass correlation is not reported for dichotomously scored items. NOTE: Cohen's Kappa is a measure of reliability that is appropriate for items that are dichotomously scored. The Intraclass Correlation Coefficient is most appropriate for items with more than two categories. Special codes assigned to student responses including blank, off-task, and not-scorable were not included in the calculation of Cohen's Kappa or intraclass correlation. SOURCE: U.S. Department of Education, Institute of Education Sciences, National Center for Education Statistics, National Assessment of Educational Progress (NAEP), 2011 Mathematics Assessment.
M3	M149901	1-3	1,500	0.98	0.98
	M150801	1-3	1,500	0.93	0.97
	M150901	1-2	1,500	0.98	†
	M151201	1-2	1,500	1.00	†
M4	M151501	1-3	1,600	0.90	0.95
	M151601	1-3	1,500	0.86	0.94
	M152602	1-5	1,500	0.95	0.97
M6	M141301	1-3	1,500	0.91	0.97
	M141601	1-2	1,500	0.75	†
	M141901	1-5	1,500	0.84	0.95
M7	M165301	1-3	1,500	0.88	0.94
	M167001	1-2	1,500	0.61	†
	M168501	1-2	1,500	0.99	†
	M168502	1-2	1,500	0.97	†
	M168503	1-3	1,500	0.76	0.90
	M168701	1-3	1,500	0.95	0.98
	M170101	1-3	1,500	0.98	0.99
M8	M164601	1-3	1,500	0.97	0.99
	M166301	1-3	1,500	0.90	0.96
	M169901	1-2	1,500	0.95	†
	M169902	1-2	1,500	0.97	†
	M169903	1-3	1,500	0.88	0.95
	M169904	1-2	1,400	0.80	†
	M170501	1-3	1,500	0.82	0.89
M9	M165001	1-2	1,500	0.92	†
	M165501	1-3	1,500	0.96	0.98
	M165701	1-3	1,500	0.96	0.96
	M165702	1-3	1,500	0.88	0.95
	M168901	1-2	1,500	0.99	†
	M170801	1-3	1,500	0.93	0.95
M11	M104901	1-3	1,500	0.94	0.96
	M162901	1-3	1,500	0.95	0.97
	M163101	1-3	1,500	0.94	0.96
	M167901	1-3	1,500	0.87	0.93
	M168301	1-3	1,500	0.92	0.97
M12	M153201	1-3	1,500	0.87	0.92
	M153601	1-2	1,500	0.99	†
	M153901	1-3	1,500	0.87	0.93

Printer-friendly Version