Table of Contents  |  Search Technical Documentation  |  References

NAEP Analysis and Scaling → Initial Activities → Constructed-Response Interrater Reliability → Range of Response Codes, Percent Agreement, and Cohen's Kappa or Intraclass Correlation for the Constructed-Response Items From Previous Years, Rescored in 2005, by Block and Item, Grade 12 Science National Main Assessment: 2005

NAEP Technical DocumentationRange of response codes, percentage exact agreement, and Cohen's Kappa or intraclass correlation for the constructed-response items from 2000, rescored in 2005, by block and item, grade 12 science national main assessment: 2005

Block	Item	Range of response codes	Sample size	Percent exact agreement	Cohen's Kappa	Intraclass correlation
S3	K049501	1–4	1,100	98	†	0.83
	K049502	1–5	1,300	87	†	0.92
	K049503	1–3	1,300	88	†	0.91
	K049504	1–3	1,300	86	†	0.90
	K049505	1–3	1,300	90	†	0.90
	K049506	1–3	1,300	85	†	0.72
S5	K040801	1–3	1,700	98	†	0.94
	K040802	1–3	1,800	98	†	0.89
	K040808	1–3	1,800	98	†	0.97
	K040809	1–3	1,600	94	†	0.95
	K040803	1–3	1,900	98	†	0.97
	K040804	1–3	2,400	86	†	0.84
	K040805	1–3	2,000	92	†	0.89
	K040806	1–2	2,200	85	0.64	†
S6	K049701	1–3	1,200	97	†	0.96
	K049702	1–3	1,200	98	†	0.97
	K049708	1–2	1,200	96	0.92	†
	K049703	1–3	1,300	93	†	0.90
	K049704	1–3	1,400	94	†	0.91
	K049705	1–4	1,000	85	†	0.88
	K049706	1–3	1,600	80	†	0.85
	K049707	1–5	2,200	80	†	0.92
† The intraclass correlation is not reported for dichotomously scored items; Cohen's Kappa is not reported for polytomously scored items. NOTE: Cohen's Kappa is a measure of reliability that is appropriate for items that are dichotomously scored. The intraclass correlation coefficient is most appropriate for items with more than two categories. SOURCE: U.S. Department of Education, Institute of Education Sciences, National Center for Education Statistics, National Assessment of Educational Progress (NAEP), 2005 Science Assessment.

Printer-friendly Version