NAEP Analysis and Scaling → Initial Activities → Constructed-Response Interrater Reliability → Range of Response Codes, Percent Agreement, and Cohen's Kappa or Intraclass Correlation for the Constructed-Response Items Used in Scaling, Grade 8 Science Combined National and State Main Assessment, by Item and Block: 2009

NAEP Technical DocumentationRange of response codes, percent agreement, and Cohen's Kappa or intraclass correlation for the constructed-response items used in scaling, grade 8 science combined national and state main assessment, by item and block: 2009

Block	Item	Range of response codes	Sample size	Percent agreement	Cohen’s Kappa	Intraclass correlation
— Not available. The intraclass correlation is not reported for dichotomously scored items; Cohen's Kappa is not reported for polytomously scored items. NOTE: Cohen's Kappa is a measure of reliability that is appropriate for items that are dichotomously scored. The Intraclass Correlation Coefficient is most appropriate for items with more than two categories. SOURCE: U.S. Department of Education, Institute of Education Sciences, National Center for Education Statistics, National Assessment of Educational Progress (NAEP), 2009 Science Assessment.
S3	K113201	1–4	1,500	95	—	0.97
	K113401	1–5	1,300	87	—	0.95
	K113701	1–5	1,400	88	—	0.94
	K113901	1–3	1,500	91	—	0.91
	K134201	1–4	1,300	97	—	0.99
S4	K114601	1–5	1,300	89	—	0.96
	K122402	1–4	1,500	89	—	0.93
	K122602	1–3	1,500	91	—	0.92
	K122801	1–3	1,100	93	—	0.90
	K125201	1–5	1,300	84	—	0.94
S5	K118501	1–4	1,500	95	—	0.96
	K120802	1–3	1,400	92	—	0.88
	K121301	1–4	1,400	96	—	0.94
	K154601	1–3	1,400	91	—	0.91
	K154701	1–5	1,300	92	—	0.91
S6	K122101	1–5	1,300	80	—	0.89
	K123801	1–4	1,400	92	—	0.95
	K123802	1–3	1,400	90	—	0.90
	K125003	1–3	1,300	93	—	0.92
	K125401	1–3	1,200	91	—	0.86
	K125402	1–3	1,300	90	—	0.90
S7	K110401	1–4	1,300	96	—	0.98
	K110501	1–3	1,300	92	—	0.89
	K110701	1–3	1,500	93	—	0.93
	K110702	1–3	1,400	94	—	0.95
	K110703	1–3	1,500	95	—	0.92
	K121001	1–5	1,200	94	—	0.93
	K121002	1–3	900	89	—	0.88
	K121003	1–3	800	90	—	0.91
	K123901	1–5	1,400	86	—	0.92
S8	K112401	1–4	1,400	98	—	0.98
	K117202	1–2	1,400	95	0.89	—
	K155601	1–5	1,400	86	—	0.83
	K155701	1–3	1,400	82	—	0.70
	K155702	1–4	1,400	86	—	0.84
S9	K114801	1–4	1,400	90	—	0.92
	K114802	1–3	1,400	92	—	0.93
	K115201	1–4	1,400	87	—	0.92
	K124102	1–4	1,400	96	—	0.98
	K156101	1–4	1,500	96	—	0.89
S10	K110901	1–3	1,400	92	—	0.86
	K115803	1–2	1,500	99	0.97	—
	K115804	1–2	1,500	100	0.99	—
	K115805	1–2	1,500	100	0.99	—
	K115901	1–3	1,100	86	—	0.85
	K115902	1–3	1,000	87	—	0.84
	K115903	1–3	900	87	—	0.72
	K116002	1–3	1,300	92	—	0.93
	K116101	1–5	1,300	88	—	0.93
	K116202	1–4	1,300	90	—	0.92
	K156401	1–3	1,500	94	—	0.86
	K156501	1–3	1,500	97	—	0.97
S11	K111801	1–3	1,300	92	—	0.91
	K111802	1–3	1,300	92	—	0.93
	K111803	1–3	1,400	94	—	0.91
	K117001	1–4	1,500	97	—	0.98
	K117003	1–2	1,400	94	0.84	—
	K117004	1–3	1,200	94	—	0.92
	K119902	1–4	1,200	91	—	0.88
	K123601	1–5	1,500	95	—	0.97
S12	K112601	1–5	1,300	93	—	0.91
	K116901	1–3	1,500	88	—	0.91
	K119505	1–3	1,500	96	—	0.95
	K123301	1–3	1,500	95	—	0.94
	K123401	1–4	1,400	90	—	0.96
	K157001	1–3	1,500	89	—	0.85

Printer-friendly Version