NAEP Analysis and Scaling → Initial Activities → Constructed-Response Interrater Reliability → Range of Response Codes, Percent Agreement, and Cohen's Kappa or Intraclass Correlation for the Constructed-Response Items Used in Scaling, Grade 4 Mathematics Combined National and State Main Assessment, by Item and Block: 2009

NAEP Technical DocumentationRange of response codes, percent agreement, and Cohen's Kappa or intraclass correlation for the constructed-response items used in scaling, grade 4 mathematics combined national and state main assessment, by item and block: 2009

Block	Item	Range of response codes	Sample size	Percent agreement	Cohen’s Kappa	Intraclass correlation
— Not available. The intraclass correlation is not reported for dichotomously scored items; Cohen's Kappa is not reported for polytomously scored items. NOTE: Cohen's Kappa is a measure of reliability that is appropriate for items that are dichotomously scored. The Intraclass Correlation Coefficient is most appropriate for items with more than two categories. SOURCE: U.S. Department of Education, Institute of Education Sciences, National Center for Education Statistics, National Assessment of Educational Progress (NAEP), 2009 Mathematics Assessment.
M3	M145901	1–3	1,700	99	—	0.99
	M146101	1–2	1,600	99	0.98	—
	M146301	1–3	1,600	98	—	0.98
	M146401	1–3	1,600	99	—	0.99
	M146601	1–5	1,600	89	—	0.97
M4	M146801	1–2	1,600	100	1.00	—
	M146901	1–3	1,700	99	—	1.00
	M147001	1–3	1,600	95	—	0.96
	M148101	1–3	1,500	92	—	0.89
M5	M019701	1–2	1,600	100	0.99	—
	M020001	1–2	1,500	99	0.98	—
	M066301	1–3	1,300	97	—	0.97
	M066501	1–3	1,300	98	—	0.98
	M067901	1–3	1,100	98	—	0.99
	M085401	1–5	1,600	93	—	0.96
	M085701	1–3	1,600	99	—	0.99
	M085901	1–3	1,600	97	—	0.97
M6	M136501	1–2	1,800	100	1.00	—
	M136601	1–3	1,700	96	—	0.97
	M136901	1–4	1,700	97	—	0.97
M7	M156701	1–2	1,700	99	0.99	—
	M157001	1–3	1,600	100	—	1.00
	M157301	1–2	1,600	100	0.99	—
	M159401	1–4	1,400	97	—	0.98
	M160701	1–3	1,600	100	—	1.00
	M162401	1–3	1,600	88	—	0.87
M8	M010631	1–3	1,800	99	—	0.99
	M040001	1–3	1,600	98	—	0.99
	M072401	1–3	1,300	97	—	0.97
	M087001	1–3	1,700	98	—	0.99
	M087301	1–5	1,100	96	—	0.98
	M091201	1–3	1,500	94	—	0.96
M9	M155101	1–5	1,600	98	—	0.98
	M157601	1–3	1,600	94	—	0.92
	M160901	1–3	1,600	98	—	0.95
	M161201	1–3	1,600	96	—	0.95
	M162001	1–3	1,600	99	—	1.00
M10	M137501	1–2	1,800	100	0.99	—
	M138201	1–2	1,800	97	0.95	—
	M138401	1–4	1,800	90	—	0.85
	M138701	1–5	1,700	94	—	0.98
M11	M156101	1–3	1,700	98	—	0.98
	M157501	1–3	1,600	99	—	0.99
	M161001	1–2	1,500	100	1.00	—
	M161002	1–3	1,500	98	—	0.98
	M161601	1–3	1,700	98	—	0.98
	M161901	1–3	1,600	98	—	0.98
	M162201	1–3	1,600	99	—	0.92
M12	M148501	1–3	1,800	99	—	1.00
	M149201	1–3	1,600	98	—	0.97
	M149301	1–2	1,700	99	0.99	—
	M149701	1–5	1,600	96	—	0.98

Printer-friendly Version