Table of Contents  |  Search Technical Documentation  |  References

NAEP Analysis and Scaling → Initial Activities → Constructed-Response Interrater Reliability → Range of Response Codes, Percentage Exact Agreement, and Cohen's Kappa or Intraclass Correlation for the Constructed-Response Items From 1994 That Were Rescored in 2001, by Block and Item, Grade 4 Geography Assessment: 2001

NAEP Technical DocumentationRange of response codes, percentage exact agreement, and Cohen's Kappa or intraclass correlation for the constructed-response items used in scaling, by block and item, grade 12 geography assessment: 2001

Block	Item	Range of response codes	Sample size	Percent exact agreement	Cohen's Kappa1	Intraclass correlation2
G4	G008503	1–3	600	94	†	0.93
	G008701	1–3	600	94	†	0.94
	G009001	1–4	600	83	†	0.89
G5	G009201	1–3	600	88	†	0.87
	G009402	1–3	600	94	†	0.91
	G009403	1–3	600	94	†	0.88
	G009601	1–4	600	91	†	0.96
G8	G012201	1–3	600	97	†	0.97
	G012503	1–3	600	99	†	0.99
	G012902	1–3	600	98	†	0.99
	G013001	1–4	600	94	†	0.97
	G013201	1–3	600	91	†	0.82
† Not applicable. The intraclass correlation is not reported for dichotomously scored items; Cohen's Kappa is not reported for polytomously scored items. 1 Cohen’s Kappa is a measure of reliability that is appropriate for items that are dichotomously scored. 2 The intraclass correlation coefficient is most appropriate for items with more than two categories. SOURCE: U.S. Department of Education, Institute of Education Sciences, National Center for Education Statistics, National Assessment of Educational Progress (NAEP), 2001 Geography Assessment.

Printer-friendly Version