NAEP Analysis and Scaling → Initial Activities → Constructed-Response Interrater Reliability → Score Range and Percent Agreement for the Constructed-Response Items From the Previous Year Assessed That Were Rescored in 2009, Grade 4 Mathematics Combined National and State Assessments, By Item and Block: 2011

NAEP Technical DocumentationScore range and percent agreement for the constructed-response items from the previous year assessed that were rescored in 2009, grade 4 mathematics combined national and state assessments, by item and block: 2011

Block	Item	Range of response codes	Sample size	Percent exact agreement
NOTE: Special codes assigned to student responses including blank, off-task, and not-scorable were included in the calculation of the Percent Exact Agreement. SOURCE: U.S. Department of Education, Institute of Education Sciences, National Center for Education Statistics, National Assessment of Educational Progress (NAEP), 2011 Mathematics Assessment.
M3	M145901	1-3	1,700	97
	M146101	1-2	1,700	96
	M146301	1-3	1,700	97
	M146401	1-3	1,700	97
	M146601	1-5	1,700	86
M4	M146801	1-2	1,700	99
	M146901	1-3	1,700	97
	M147001	1-3	1,700	94
	M148101	1-3	1,700	90
M6	M136501	1-2	1,600	98
	M136601	1-3	1,600	94
	M136901	1-3	1,600	95
M7	M156701	1-2	1,600	99
	M157001	1-3	1,600	99
	M157301	1-2	1,600	98
	M159401	1-4	1,600	95
	M160701	1-3	1,600	99
	M162401	1-3	1,600	85
M8	M010631	1-3	1,600	99
	M040001	1-3	1,800	98
	M072401	1-3	1,600	93
	M087001	1-3	1,600	98
	M087301	1-5	1,600	93
	M091201	1-3	1,600	93
M9	M155101	1-5	1,600	95
	M157601	1-3	1,600	88
	M160901	1-3	1,600	92
	M161201	1-3	1,600	96
	M162001	1-3	1,600	99
M11	M156101	1-3	1,700	98
	M157501	1-3	1,700	97
	M161001	1-2	1,700	99
	M161002	1-3	1,700	95
	M161601	1-3	1,700	92
	M161901	1-3	1,700	96
	M162201	1-3	1,700	97
M12	M148501	1-3	1,600	98
	M149201	1-3	1,600	96
	M149301	1-2	1,600	96
	M149701	1-5	1,600	93

Score range and Cohen's Kappa or intraclass correlation for the constructed-response items from the previous year assessed that were rescored in 2009, grade 4 mathematics combined national and state assessments, by item and block: 2011

Block	Item	Range of response codes	Sample size	Cohen’s Kappa	Intraclass correlation
† Not applicable. The intraclass correlation is not reported for dichotomously scored items. NOTE: Cohen's Kappa is a measure of reliability that is appropriate for items that are dichotomously scored. The Intraclass Correlation Coefficient is most appropriate for items with more than two categories. Special codes assigned to student responses including blank, off-task, and not-scorable were not included in the calculation of Cohen's Kappa or intraclass correlation. SOURCE: U.S. Department of Education, Institute of Education Sciences, National Center for Education Statistics, National Assessment of Educational Progress (NAEP), 2011 Mathematics Assessment.
M3	M145901	1-3	1,500	0.95	0.98
	M146101	1-2	1,500	0.97	†
	M146301	1-3	1,500	0.95	0.98
	M146401	1-3	1,500	0.95	0.96
	M146601	1-5	1,500	0.80	0.95
M4	M146801	1-2	1,500	0.99	†
	M146901	1-3	1,500	0.97	0.98
	M147001	1-3	1,500	0.91	0.94
	M148101	1-3	1,500	0.76	0.86
M6	M136501	1-2	1,500	0.98	†
	M136601	1-3	1,500	0.91	0.95
	M136901	1-3	1,500	0.92	0.96
M7	M156701	1-2	1,500	0.99	†
	M157001	1-3	1,500	0.99	0.99
	M157301	1-2	1,500	0.98	†
	M159401	1-4	1,500	0.91	0.97
	M160701	1-3	1,500	0.99	0.99
	M162401	1-3	1,500	0.70	0.83
M8	M010631	1-3	1,500	0.98	0.99
	M040001	1-3	1,600	0.97	0.99
	M072401	1-3	1,500	0.89	0.95
	M087001	1-3	1,500	0.97	0.99
	M087301	1-5	1,400	0.90	0.98
	M091201	1-3	1,500	0.87	0.95
M9	M155101	1-5	1,500	0.92	0.97
	M157601	1-3	1,500	0.71	0.81
	M160901	1-3	1,500	0.76	0.86
	M161201	1-3	1,500	0.91	0.95
	M162001	1-3	1,500	0.99	0.99
M11	M156101	1-3	1,500	0.96	0.98
	M157501	1-3	1,500	0.97	0.97
	M161001	1-2	1,500	0.98	†
	M161002	1-3	1,500	0.90	0.96
	M161601	1-3	1,500	0.92	0.95
	M161901	1-3	1,500	0.94	0.96
	M162201	1-3	1,500	0.90	0.91
M12	M148501	1-3	1,500	0.97	0.99
	M149201	1-3	1,500	0.91	0.95
	M149301	1-2	1,500	0.93	†
	M149701	1-5	1,500	0.90	0.97

Printer-friendly Version