Table of Contents  |  Search Technical Documentation  |  References

NAEP Analysis and Scaling → Initial Activities → Constructed-Response Interrater Reliability → Range of Response Codes, Percent Exact Agreement, and Cohen's Kappa or Intraclass Correlation for the Constructed-Response Items Used in Scaling, by Block and Item, Age 17 Reading Long-Term Trend Bridge Study: 2004

NAEP Technical DocumentationRange of response codes, percent exact agreement, and Cohen's Kappa or intraclass correlation for the constructed-response items used in scaling, by block and item, age 17 reading long-term trend bridge study: 2004

Block	Item	Range of response codes	Sample size	Percent exact agreement	Cohen's Kappa	Intraclass correlation
H	N001527	1–2	200	98	0.93	†
J	N001904	1–2	200	96	0.89	†
L	N002804	1–2	200	94	0.85	†
M	N003104	1–2	200	90	0.76	†
N	N003704	1–2	200	95	0.90	†
O	N004303	1–2	100	92	0.83	†
P	N004605	1–2	200	97	0.90	†
Q	N015905	1–2	100	93	0.85	†
R21	N021203	1–5	200	89	†	0.92
R22	N023802	1–4	200	88	†	0.83
R23	N021803	1–4	200	93	†	0.95
† The intraclass correlation is not reported for dichotomously scored items; Cohen's Kappa is not reported for polytomously scored items. NOTE: Cohen's Kappa is a measure of reliability that is appropriate for items that are dichotomously scored. The intraclass correlation coefficient is most appropriate for items with more than two categories. SOURCE: U.S. Department of Education, Institute of Education Sciences, National Center for Education Statistics, National Assessment of Educational Progress (NAEP), 2004 Reading Long-Term Trend Assessment.

Printer-friendly Version