Table of Contents  |  Search Technical Documentation  |  References

NAEP Analysis and Scaling → Initial Activities → Constructed-Response Interrater Reliability → Range of Response Codes, Percentage Exact Agreement, and Cohen's Kappa or Intraclass Correlation for the Constructed-Response Items Used in Scaling, by Block and Item, Grade 4 Science Assessment: 2000

NAEP Technical DocumentationRange of response codes, percentage exact agreement, and Cohen's Kappa or intraclass correlation for the constructed-response items used in scaling, by block and item, grade 4 science assessment: 2000

Block	Item	Range of response codes	Sample size	Percentage exact agreement	Cohen's Kappa	Intraclass correlation
S3	K031001	1?3	1,100	96	?	0.94
	K031002	1?3	1,100	91	?	0.86
	K031003	1?3	1,100	88	?	0.78
	K031004	1?2	1,100	91	0.84	?
	K031005	1?3	1,100	89	?	0.87
	K031006	1?3	1,100	94	?	0.97
	K031007	1?3	1,100	90	?	0.94
S4	K031101	1?2	1,100	98	0.95	?
	K031102	1?2	1,100	97	0.94	?
	K031103	2-2	1,100	96	0.93	?
	K031104	3-2	1,100	99	0.97	?
	K031105	1?3	1,100	100	?	1.00
	K031107	1?4	1,100	96	?	0.98
S6	K031301	1?4	1,200	94	?	0.97
	K031302	1?3	1,200	95	?	0.93
	K031303	1?3	1,200	94	?	0.98
	K031304	1?3	1,200	97	?	0.99
	K031309	1?4	1,200	92	?	0.97
S7	K031401	1?4	1,500	89	?	0.89
	K031402	1?3	1,500	93	?	0.95
	K031403	1?3	1,500	94	?	0.95
	K031404	1?3	1,500	97	?	0.99
	K031407	1?3	1,500	94	?	0.94
	K031408	1?3	1,500	98	?	0.99
	K031409	1?3	1,500	98	?	0.98
	K031410	1?3	1,500	97	?	0.98
S9	K031602	1?3	1,500	98	?	0.99
	K031603	1?3	1,500	99	?	0.99
	K031604	1?3	1,500	99	?	0.98
	K031606	1?3	1,500	96	?	0.95
	K031607	1?4	1,500	95	?	0.97
	K031608	1?3	1,500	96	?	0.95
	K031609	1?3	1,500	99	?	0.98
S10	K031901	1?3	1,100	95	?	0.96
	K032001	1?3	1,100	97	?	0.97
	K032501	1?3	1,100	93	?	0.95
	K032502	1?3	1,100	92	?	0.93
	K032601	1?3	1,100	96	?	0.95
	K032602	1?3	1,100	96	?	0.97
S12	K034001	1?3	1,100	97	?	0.93
	K034101	1?3	1,100	96	?	0.96
	K034401	1?3	1,100	95	?	0.98
	K034501	1?4	1,100	93	?	0.97
	K034502	1?3	1,100	99	?	0.99
S13	K034802	1?3	1,200	93	?	0.90
	K034901	1?3	1,200	93	?	0.92
	K034902	1?3	1,200	93	?	0.94
	K035201	1?3	1,200	94	?	0.96
	K035301	1?4	1,200	95	?	0.97
S14	K035601	1?3	1,500	92	?	0.89
	K035801	1?3	1,500	93	?	0.95
	K035901	1?3	1,500	98	?	0.96
	K036101	1?3	1,500	96	?	0.97
	K036301	1?3	1,500	97	?	0.97
S15	K037301	1?3	1,100	98	?	0.96
	K037401	1?3	1,100	96	?	0.88
	K037501	1?3	1,200	99	?	0.97
	K037601	1?3	1,100	96	?	0.96
	K037701	1?4	1,100	98	?	0.97
	K037702	1?3	1,100	98	?	0.99
S21	K039801	1?3	1,200	97	?	0.95
	K039901	1?5	1,200	93	?	0.97
	K040001	1?3	1,200	93	?	0.96
	K040301	1?3	1,200	93	?	0.94
	K040401	1?4	1,200	95	?	0.93
	K040501	1?3	1,200	99	?	0.99
S11	K092201	1?3	1,100	95	?	0.96
S20	K096901	1?3	1,200	94	?	0.92
S11	K098201	1?3	1,200	95	?	0.94
	K098301	1?3	1,100	94	?	0.91
S20	K098401	1?3	1,200	97	?	0.96
S11	K099501	1?3	1,100	97	?	0.97
S20	K099601	1?3	1,200	92	?	0.95
? The intraclass correlation is not reported for dichotomously scored items; Cohen's Kappa is not reported for polytomously scored items. NOTE: Cohen's Kappa is a measure of reliability that is appropriate for items that are dichotomously scored. The intraclass correlation coefficient is most appropriate for items with more than two categories. SOURCE: U.S. Department of Education, Institute of Education Sciences, National Center for Education Statistics, National Assessment of Educational Progress (NAEP), 2000 Science Assessment.

Printer-friendly Version