NAEP Analysis and Scaling → Initial Activities → Constructed-Response Interrater Reliability → Range of Response Codes, Percent Agreement, and Cohen's Kappa or Intraclass Correlation for the Constructed-Response Items Used in Scaling, Grade 4 Geography National Assessment, by Item and Block: 2010

NAEP Technical DocumentationRange of response codes, percent agreement, and Cohen's Kappa or intraclass correlation for the constructed-response items used in scaling, grade 4 geography national assessment, by item and block: 2010

Block	Item	Range of response codes	Sample size	Percent exact agreement	Cohen’s Kappa	Intraclass correlation
† The intraclass correlation is not reported for dichotomously scored items; Cohen's Kappa is not reported for polytomously scored items. NOTE: Cohen's Kappa is a measure of reliability that is appropriate for items that are dichotomously scored. The intraclass correlation coefficient is most appropriate for items with more than two categories. SOURCE: U.S. Department of Education, Institute of Education Sciences, National Center for Education Statistics, National Assessment of Educational Progress (NAEP), 2010 Geography Assessment.
G3	G027201	1-2	500	99	0.98	†
	G027301	1-2	500	99	0.97	†
	G027401	1-2	500	98	0.95	†
	G027501	1-2	500	99	0.98	†
	G027601	1-2	400	100	0.99	†
	G027701	1-2	500	98	0.96	†
	G027801	1-2	500	97	0.95	†
	G028201	1-4	500	80	†	0.89
	G028401	1-3	600	75	†	0.71
	G028501	1-3	400	92	†	0.91
	G028701	1-4	500	81	†	0.89
G4	G008001	1-3	400	95	†	0.94
	G008201	1-4	500	89	†	0.91
	G008503	1-3	500	92	†	0.91
	G008701	1-3	500	93	†	0.94
	G009001	1-4	600	83	†	0.89
G5	G009201	1-3	600	88	†	0.88
	G009402	1-3	600	94	†	0.90
	G009403	1-3	600	95	†	0.89
	G009601	1-4	600	91	†	0.94
G6	G029301	1-2	600	98	0.96	†
	G029401	1-2	500	99	0.97	†
	G029501	1-2	600	99	0.97	†
	G029601	1-2	500	99	0.87	†
	G029701	1-2	600	95	0.82	†
	G030101	1-3	600	76	†	0.57
	G030501	1-4	600	82	†	0.89
	G030801	1-2	500	100	0.99	†
	G030901	1-2	500	98	0.97	†
	G031001	1-2	500	98	0.96	†
	G031101	1-2	500	99	0.98	†
	G031401	1-4	500	90	†	0.94
G7	G043701	1-4	600	99	†	1.00
	G043704	1-3	500	95	†	0.93
	G044202	1-2	400	99	0.99	†
	G045203	1-3	400	89	†	0.87
G8	G041701	1-3	500	89	†	0.88
	G042601	1-3	500	98	†	0.97
	G042702	1-2	500	96	0.92	†
	G043201	1-3	500	83	†	0.83
	G043401	1-3	500	92	†	0.88
	G059401	1-4	600	98	†	0.98

Printer-friendly Version