Table of Contents  |  Search Technical Documentation  |  References

NAEP Analysis and Scaling → Initial Activities → Constructed-Response Interrater Reliability → Range of Response Codes, Percent Exact Agreement, and Cohen's Kappa or Intraclass Correlation for the Constructed-Response Items Used in Scaling, by Block and Item, Age 17 Mathematics Long-Term Trend Assessment: 2004

NAEP Technical DocumentationRange of response codes, percentage exact agreement, and Cohen's Kappa or intraclass correlation for the constructed-response items used in scaling, by block and item, age 17 mathematics long-term trend assessment: 2004

Block	Item	Range of response codes	Sample size	Percent exact agreement	Cohen's Kappa
M21	N307301	1–2	900	99	0.97
	N307701	1–2	900	100	0.99
	N308101	1–2	900	99	0.98
	N308501	1–2	900	98	0.96
M22	N302901	1–2	900	99	0.93
	N308901	1–2	900	99	0.98
	N309301	1–2	900	99	0.97
	N309901	1–2	900	99	0.98
	N310201	1–2	900	100	0.99
M23	N316701	1–2	900	100	0.98
	N318901	1–2	900	99	0.97
	N319301	1–2	900	100	1.00
	N319302	1–2	900	99	0.99
	N319303	1–2	900	100	1.00
	N320101	1–2	900	100	1.00
	N320301	1–2	800	98	0.97
	N320302	1–2	800	99	0.97
	N320401	1–2	800	100	0.99
	N320601	1–2	800	99	0.97
M24	N315501	1–2	800	99	0.98
	N321001	1–2	900	99	0.99
	N321101	1–2	800	100	1.00
	N321401	1–2	700	100	0.99
	N321901	1–2	700	100	0.99
MX51	N303801	1–2	900	100	0.99
	N304101	1–2	900	100	1.00
	N304201	1–2	900	99	0.96
	N304601	1–2	900	99	0.99
	N304901	1–2	900	100	1.00
MX52	N299001	1–2	900	99	0.98
	N305501	1–2	900	99	0.99
	N305801	1–2	900	100	0.99
	N306201	1–2	800	98	0.96
	N306401	1–2	900	100	0.99
NOTE: Cohen's Kappa is a measure of reliability that is appropriate for items that are dichotomously scored. SOURCE: U.S. Department of Education, Institute of Education Sciences, National Center for Education Statistics, National Assessment of Educational Progress (NAEP), 2004 Mathematics Long-Term Trend Assessment.

Printer-friendly Version