Table of Contents  |  Search Technical Documentation  |  References

NAEP Analysis and Scaling → Initial Activities → Constructed-Response Interrater Reliability → Range of Response Codes, Percent Exact Agreement, and Cohen's Kappa or Intraclass Correlation for the Constructed-Response Items From Previous Years, Rescored in 2005, by Block and Item, Grade 4 Science Combined National and State Main Assessment: 2005

NAEP Technical DocumentationRange of response codes, percentage exact agreement, and Cohen's Kappa or intraclass correlation for the constructed-response items from 2000, rescored in 2005, by block and item, grade 4 science combined national and state main assessment: 2005

Block	Item	Range of response codes	Sample size	Percent exact agreement	Cohen's Kappa	Intraclass correlation
S3	K031001	1–3	3,500	94	†	0.88
	K031002	1–3	3,900	85	†	0.78
	K031003	1–3	3,700	83	†	0.65
	K031004	1–2	4,200	88	0.74	†
	K031005	1–3	4,000	78	†	0.68
	K031006	1–3	3,400	88	†	0.89
	K031007	1–3	4,100	83	†	0.85
S4	K031101	1–2	3,100	97	0.88	†
	K031102	1–2	2,500	95	0.87	†
	K031103	1–2	2,700	93	0.86	†
	K031104	1–2	2,900	98	0.90	†
	K031105	1–3	2,800	99	†	0.97
	K031107	1–4	2,900	92	†	0.95
S6	K031301	1–4	2,800	91	†	0.87
	K031309	1–4	2,500	90	†	0.90
	K031302	1–3	2,800	89	†	0.54
	K031303	1–3	2,400	89	†	0.91
	K031304	1–3	3,500	92	†	0.92
† The intraclass correlation is not reported for dichotomously scored items; Cohen's Kappa is not reported for polytomously scored items. NOTE: Cohen's Kappa is a measure of reliability that is appropriate for items that are dichotomously scored. The intraclass correlation coefficient is most appropriate for items with more than two categories. SOURCE: U.S. Department of Education, Institute of Education Sciences, National Center for Education Statistics, National Assessment of Educational Progress (NAEP), 2005 Science Assessment.

Printer-friendly Version