NAEP Analysis and Scaling → Initial Activities → Constructed-Response Interrater Reliability → Range of Response Codes, Percent Agreement, and Cohen's Kappa or Intraclass Correlation for the Constructed-Response Items Used in Scaling, Grade 12 Science National Main Assessment, by Item and Block: 2009

NAEP Technical DocumentationRange of response codes, percent agreement, and Cohen's Kappa or intraclass correlation for the constructed-response items used in scaling, grade 12 science national main assessment, by item and block: 2009

Block	Item	Range of response codes	Sample size	Percent agreement	Cohen’s Kappa	Intraclass correlation
— Not available. The intraclass correlation is not reported for dichotomously scored items; Cohen's Kappa is not reported for polytomously scored items. NOTE: Cohen's Kappa is a measure of reliability that is appropriate for items that are dichotomously scored. The Intraclass Correlation Coefficient is most appropriate for items with more than two categories. SOURCE: U.S. Department of Education, Institute of Education Sciences, National Center for Education Statistics, National Assessment of Educational Progress (NAEP), 2009 Science Assessment.
S3	K126202	1–3	500	96	—	0.93
	K126303	1–5	500	93	—	0.95
	K126304	1–3	500	93	—	0.90
	K133301	1–5	500	93	—	0.93
	K139001	1–3	400	87	—	0.81
	K157201	1–3	500	88	—	0.89
	K157202	1–3	400	92	—	0.90
S4	K128403	1–4	300	88	—	0.83
	K133801	1–5	500	83	—	0.90
	K137201	1–4	500	91	—	0.91
	K144401	1–4	500	91	—	0.90
	K157901	1–3	400	96	—	0.96
S5	K129601	1–3	400	93	—	0.91
	K130301	1–3	500	92	—	0.87
	K131001	1–4	500	92	—	0.85
	K143903	1–3	400	95	—	0.92
	K158001	1–4	500	97	—	0.97
S6	K134503	1–3	400	88	—	0.83
	K135001	1–4	400	93	—	0.91
	K139801	1–3	500	92	—	0.92
	K143801	1–3	400	98	—	0.92
	K158201	1–3	500	91	—	0.93
	K158202	1–3	500	95	—	0.96
	K158203	1–3	500	92	—	0.91
	K158204	1–3	500	96	—	0.92
S7	K132701	1–3	400	95	—	0.89
	K135401	1–3	400	94	—	0.96
	K136101	1–4	500	97	—	0.97
	K136102	1–3	500	94	—	0.95
	K136103	1–4	500	97	—	0.97
	K158301	1–3	400	91	—	0.86
S8	K127201	1–5	500	89	—	0.93
	K136401	1–3	400	97	—	0.97
	K137102	1–4	400	94	—	0.97
	K137601	1–5	500	97	—	0.99
	K142401	1–3	500	95	—	0.96
	K158401	1–3	400	88	—	0.83
	K158402	1–3	300	93	—	0.90
S9	K127901	1–3	300	91	—	0.88
	K131103	1–3	500	93	—	0.94
	K138401	1–4	400	99	—	1.00
	K138402	1–4	500	94	—	0.90
	K139101	1–3	400	96	—	0.97
	K142601	1–5	500	85	—	0.83
S10	K137701	1–4	500	96	—	0.96
	K141201	1–5	300	98	—	1.00
	K141503	1–3	400	93	—	0.86
	K158501	1–3	400	87	—	0.72
	K158502	1–3	400	91	—	0.80
	K158601	1–3	400	95	—	0.92
	K158602	1–3	400	89	—	0.85
	K158603	1–3	400	95	—	0.93
S11	K125802	1–3	400	87	—	0.80
	K125803	1–3	400	98	—	0.98
	K127701	1–4	500	89	—	0.91
	K141701	1–4	500	97	—	0.98
	K144202	1–3	400	83	—	0.76
S12	K128701	1–5	500	66	—	0.81
	K129001	1–3	400	96	—	0.80
	K130001	1–4	500	92	—	0.85
	K130002	1–4	500	94	—	0.94
	K133401	1–3	400	88	—	0.84
S13	K126001	1–5	500	83	—	0.91
	K138501	1–4	300	98	—	0.98
	K140301	1–3	500	94	—	0.93
	K140303	1–3	400	91	—	0.94
	K158701	1–3	500	99	—	0.98
	K158702	1–3	500	98	—	0.98

Printer-friendly Version