Table of Contents  |  Search Technical Documentation  |  References

NAEP Analysis and Scaling → Initial Activities → Constructed-Response Interrater Reliability → Range of Response Codes, Percentage Exact Agreement, and Cohen's Kappa or Intraclass Correlation for the Constructed-Response Items Used in Scaling, by Block and Item, Grade 4 Geography Assessment: 2001

NAEP Technical DocumentationRange of response codes, percentage exact agreement, and Cohen's Kappa or intraclass correlation for the constructed-response items used in scaling, by block and item, grade 4 geography assessment: 2001

Block	Item	Range of response codes	Sample size	Percent exact agreement	Cohen's Kappa	Intraclass correlation
G3	G027201	1–2	700	100	0.99	†
	G027301	1–2	700	99	0.99	†
	G027401	1–2	700	95	0.93	†
	G027501	1–2	700	98	0.97	†
	G027601	1–2	700	97	0.96	†
	G027701	1–2	700	94	0.91	†
	G027801	1–2	700	96	0.94	†
	G028201	1–4	700	91	†	0.97
	G028401	1–3	700	89	†	0.93
	G028501	1–3	700	94	†	0.95
	G028701	1–4	700	91	†	0.95
G4	G008001	1–3	600	99	†	1.00
	G008201	1–4	600	93	†	0.97
	G008503	1–3	600	94	†	0.96
	G008701	1–3	600	94	†	0.97
	G009001	1–4	600	84	†	0.95
G5	G009201	1–3	600	95	†	0.97
	G009402	1–3	600	96	†	0.95
	G009403	1–3	600	95	†	0.93
	G009601	1–4	600	94	†	0.98
G6	G029301	1–2	700	97	0.96	†
	G029401	1–2	700	98	0.97	†
	G029501	1–2	700	98	0.96	†
	G029601	1–2	700	99	0.98	†
	G029701	1–2	700	98	0.95	†
	G030101	1–3	700	87	†	0.85
	G030501	1–4	700	81	†	0.93
	G030801	1–2	700	98	0.98	†
	G030901	1–2	700	99	0.99	†
	G031001	1–2	700	98	0.97	†
	G031101	1–2	700	98	0.98	†
	G031401	1–4	700	96	†	0.99
G7	G031801	1–3	600	96	†	0.97
	G031901	1–3	600	95	†	0.97
	G032401	1–3	600	100	†	1.00
	G032601	1–2	600	98	0.97	†
G8	G012201	1–3	600	98	†	0.99
	G012503	1–3	600	100	†	1.00
	G012902	1–3	600	99	†	0.99
	G013001	1–4	600	94	†	0.97
	G013201	1–3	600	93	†	0.92
† The intraclass correlation is not reported for dichotomously scored items; Cohen's Kappa is not reported for polytomously scored items. NOTE: Cohen's Kappa is a measure of reliability that is appropriate for items that are dichotomously scored. The intraclass correlation coefficient is most appropriate for items with more than two categories. SOURCE: U.S. Department of Education, Institute of Education Sciences, National Center for Education Statistics, National Assessment of Educational Progress (NAEP), 2001 Geography Assessment.

Printer-friendly Version