Table of Contents  |  Search Technical Documentation  |  References

NAEP Analysis and Scaling → Initial Activities → Constructed-Response Interrater Reliability → Range of Response Codes, Percent Exact Agreement, and Cohen's Kappa or Intraclass Correlation for the Constructed-Response Items Used in Scaling, by Block and Item, Grade 4 Science Combined National and State Main Assessment: 2005

NAEP Technical DocumentationRange of response codes, percentage exact agreement, and Cohen's Kappa or intraclass correlation for the constructed-response items used in scaling, by block and item, grade 4 science combined national and state main assessment: 2005

Block	Item	Range of response codes	Sample size	Percent exact agreement	Cohen's Kappa	Intraclass correlation
B3	K031001	1–3	1,300	98	†	0.96
	K031002	1–3	1,300	95	†	0.94
	K031003	1–3	1,300	94	†	0.90
	K031004	1–2	1,200	95	0.90	†
	K031005	1–3	1,200	92	†	0.88
	K031006	1–3	1,100	95	†	0.96
	K031007	1–3	1,000	92	†	0.93
B4	K031101	1–2	1,300	98	0.92	†
	K031102	1–2	1,300	96	0.92	†
	K031103	1–2	1,300	94	0.89	†
	K031104	1–2	1,200	98	0.93	†
	K031105	1–3	1,200	98	†	0.96
	K031107	1–4	1,100	95	†	0.96
B6	K031301	1–4	1,200	93	†	0.91
	K031302	1–3	1,000	90	†	0.45
	K031303	1–3	900	87	†	0.89
	K031304	1–3	900	91	†	0.91
	K031309	1–4	1,100	92	†	0.93
B7	K031401	1–4	1,400	87	†	0.85
	K031402	1–3	1,400	88	†	0.88
	K031403	1–3	1,400	88	†	0.89
	K031404	1–3	1,200	94	†	0.94
	K031407	1–3	1,400	90	†	0.89
	K031408	1–3	1,200	95	†	0.94
	K031409	1–3	1,400	91	†	0.86
	K031410	1–3	1,400	93	†	0.93
B8	K103101	1–3	1,400	92	†	0.86
	K103901	1–3	1,300	92	†	0.87
B10	K031901	1–3	1,400	90	†	0.90
	K032001	1–3	1,400	91	†	0.91
	K032501	1–3	1,400	92	†	0.94
	K032502	1–3	1,400	92	†	0.91
	K032601	1–3	1,400	91	†	0.87
	K032602	1–3	1,300	92	†	0.89
B11	K092201	1–3	1,300	92	†	0.90
	K098201	1–3	1,400	90	†	0.84
	K098301	1–3	1,400	88	†	0.79
	K099501	1–3	1,400	94	†	0.93
B13	K034802	1–3	1,400	89	†	0.84
	K034901	1–3	1,400	96	†	0.95
	K034902	1–3	1,400	94	†	0.90
	K035201	1–3	1,400	92	†	0.93
	K035301	1–4	1,400	93	†	0.91
B14	K035601	1–3	1,400	93	†	0.88
	K035801	1–3	1,400	92	†	0.94
	K035901	1–3	1,300	98	†	0.94
	K036101	1–3	1,300	91	†	0.88
	K036301	1–3	1,300	98	†	0.96
B15	K037301	1–3	1,400	97	†	0.95
	K037401	1–3	1,100	95	†	0.73
	K037601	1–3	1,300	90	†	0.83
	K037701	1–4	1,400	98	†	0.97
	K037702	1–3	1,300	89	†	0.84
B20	K096901	1–3	1,500	93	†	0.94
	K098401	1–3	1,400	96	†	0.93
	K099601	1–3	1,400	93	†	0.92
	K100701	1–3	1,400	96	†	0.97
S9	K060203	1–3	1,300	88	†	0.77
	K060204	1–3	1,500	93	†	0.93
	K060206	1–3	1,500	90	†	0.87
	K060207	1–4	1,400	86	†	0.92
	K060301	1–3	1,400	93	†	0.89
S10	K060401	1–4	1,300	90	†	0.91
S12	K060901	1–3	1,300	86	†	0.84
S14	K061601	1–3	1,300	97	†	0.97
S15	K062001	1–4	1,400	88	†	0.86
	K062201	1–4	1,500	91	†	0.94
	K062401	1–3	1,400	92	†	0.93
	K062903	1–4	1,300	95	†	0.96
	K063001	1–3	1,100	94	†	0.89
† The intraclass correlation is not reported for dichotomously scored items; Cohen's Kappa is not reported for polytomously scored items. NOTE: Cohen's Kappa is a measure of reliability that is appropriate for items that are dichotomously scored. The intraclass correlation coefficient is most appropriate for items with more than two categories. SOURCE: U.S. Department of Education, Institute of Education Sciences, National Center for Education Statistics, National Assessment of Educational Progress (NAEP), 2005 Science Assessment.

Printer-friendly Version