Table of Contents  |  Search Technical Documentation  |  References

NAEP Analysis and Scaling → Initial Activities → Constructed-Response Interrater Reliability → Range of Response Codes, Percent Exact Agreement, and Cohen's Kappa or Intraclass Correlation for the Constructed-Response Items From Previous Years, Rescored in 2005, by Block and Item, Grade 4 Mathematics Combined National and State Main Assessment: 2005

NAEP Technical DocumentationRange of response codes, percentage exact agreement, and Cohen's Kappa or intraclass correlation for the constructed-response items from previous years,  rescored in 2005, by block and item, grade 4 mathematics combined national and state main assessment: 2005

Block	Item	Range of response codes	Sample size	Percent exact agreement	Cohen's Kappa	Intraclass correlation
M3	M019901	1–3	400	99	†	0.98
	M020101	1–2	600	99	0.98	†
	M043501	1–5	900	91	†	0.96
	M047301	1–4	800	97	†	0.98
	M074301	1–2	800	99	0.99	†
	M074501	1–3	2,200	89	†	0.90
	M074801	1–3	500	98	†	0.98
	M074901	1–3	2,000	92	†	0.94
	M075001	1–3	700	98	†	0.98
	N277903	1–2	400	98	0.96	†
M4	M046001	1–5	400	98	†	0.98
	M046601	1–4	2,200	91	†	0.91
	M046801	1–5	800	98	†	0.97
	M046901	1–5	400	99	†	0.99
	M066601	1–3	1,200	90	†	0.91
	M068001	1–3	600	97	†	0.96
	M068002	1–3	400	97	†	0.97
	M068003	1–3	800	97	†	0.95
	M068004	1–5	1,800	88	†	0.96
M5	M019701	1–2	800	99	0.98	†
	M020001	1–2	800	98	0.96	†
	M066301	1–3	1,700	95	†	0.92
	M066501	1–3	1,500	96	†	0.97
	M067901	1–3	700	96	†	0.96
	M085401	1–5	1,500	86	†	0.95
	M085701	1–3	600	93	†	0.93
	M085901	1–3	3,300	94	†	0.94
M6	M136501	1–2	400	99	0.98	†
	M136601	1–3	700	95	†	0.96
	M136901	1–4	1,300	93	†	0.97
M7	M139301	1–2	1,000	99	0.98	†
	M139601	1–3	400	99	†	1.00
	M139801	1–4	900	94	†	0.97
	M140001	1–3	500	97	†	0.98
	M140301	1–5	1,300	94	†	0.98
M8	M010631	1–3	400	96	†	0.93
	M040001	1–3	1,000	97	†	0.98
	M072401	1–3	600	94	†	0.95
	M087001	1–3	600	94	†	0.96
	M087301	1–5	900	92	†	0.95
	M091201	1–3	2,200	89	†	0.91
M9	M040201	1–2	800	91	0.72	†
	M043201	1–2	600	98	0.96	†
	M043301	1–3	400	99	†	0.99
	M043401	1–4	400	95	†	0.98
	M043402	1–4	1,000	96	†	0.97
	M043403	1–3	400	96	†	0.96
	M066701	1–3	700	93	†	0.95
	M066801	1–3	600	95	†	0.95
	M086601	1–3	800	96	†	0.95
	M091401	1–5	3,900	84	†	0.95
M10	M137501	1–2	400	99	0.97	†
	M138201	1–2	900	96	0.92	†
	M138401	1–4	1,600	86	†	0.95
	M138701	1–5	700	88	†	0.96
M11	M096001	1–2	1,000	96	0.91	†
	M097201	1–3	1,000	94	†	0.96
	M097401	1–3	600	97	†	0.95
M12	M100201	1–2	400	99	0.98	†
	M100501	1–2	400	100	1.00	†
	M101401	1–2	400	99	0.97	†
	M101501	1–3	600	98	†	0.97
† The intraclass correlation is not reported for dichotomously scored items; Cohen's Kappa is not reported for polytomously scored items. NOTE: Cohen's Kappa is a measure of reliability that is appropriate for items that are dichotomously scored. The intraclass correlation coefficient is most appropriate for items with more than two categories. Sample sizes are rounded the nearest hundred. SOURCE: U.S. Department of Education, Institute of Education Sciences, National Center for Education Statistics, National Assessment of Educational Progress (NAEP), 2005 Mathematics Assessment.

Printer-friendly Version