NAEP Analysis and Scaling → Initial Activities → Constructed-Response Interrater Reliability → Range of Response Codes, Percent Agreement, and Cohen's Kappa or Intraclass Correlation for the Constructed-Response Items Used in Scaling, Grade 4 Science Combined National and State Main Assessment, by Item and Block: 2009

NAEP Technical DocumentationRange of response codes, percent agreement, and Cohen's Kappa or intraclass correlation for the constructed-response items used in scaling, grade 4 science combined national and state main assessment, by item and block: 2009

Block	Item	Range of response codes	Sample size	Percent agreement	Cohen’s Kappa	Intraclass correlation
— Not available. The intraclass correlation is not reported for dichotomously scored items; Cohen's Kappa is not reported for polytomously scored items. NOTE: Cohen's Kappa is a measure of reliability that is appropriate for items that are dichotomously scored. The Intraclass Correlation Coefficient is most appropriate for items with more than two categories. SOURCE: U.S. Department of Education, Institute of Education Sciences, National Center for Education Statistics, National Assessment of Educational Progress (NAEP), 2009 Science Assessment.
S3	K077602	1–4	1,900	90	—	0.94
	K080404	1–3	1,900	92	—	0.90
	K106701	1–3	1,400	83	—	0.81
	K153001	1–3	1,700	87	—	0.82
	K153102	1–4	1,500	90	—	0.93
S4	K078902	1–3	1,800	97	—	0.94
	K079501	1–4	1,600	93	—	0.95
	K084301	1–2	1,800	95	0.90	—
	K084302	1–2	1,700	95	0.90	—
	K153301	1–3	1,800	92	—	0.92
S5	K081601	1–3	1,800	96	—	0.91
	K081801	1–3	1,700	91	—	0.88
	K082501	1–3	1,500	91	—	0.90
	K084202	1–3	1,300	95	—	0.81
	K087301	1–3	1,800	92	—	0.91
S6	K076501	1–3	1,700	91	—	0.91
	K086901	1–2	1,600	100	0.99	—
	K086902	1–2	1,700	100	1.00	—
	K086903	1–2	1,700	100	0.99	—
	K086904	1–2	1,600	100	1.00	—
	K087402	1–4	1,700	91	—	0.90
	K107902	1–3	1,800	93	—	0.93
	K109102	1–3	1,500	87	—	0.80
S7	K076302	1–3	1,700	93	—	0.88
	K076901	1–3	1,500	93	—	0.93
	K077201	1–3	1,200	91	—	0.93
	K080703	1–4	1,700	95	—	0.92
	K087101	1–3	1,400	96	—	0.96
	K153601	1–3	1,500	95	—	0.96
S8	K077501	1–3	1,700	90	—	0.90
	K077701	1–3	1,700	96	—	0.97
	K078102	1–4	1,600	92	—	0.94
	K078301	1–3	1,100	95	—	0.95
	K109701	1–3	1,300	91	—	0.84
	K109702	1–3	1,200	91	—	0.78
	K109703	1–3	1,100	89	—	0.81
S9	K088302	1–3	1,700	98	—	0.97
	K088401	1–5	1,700	86	—	0.92
	K109402	1–3	1,500	99	—	0.98
	K153701	1–3	1,500	91	—	0.89
	K153702	1–3	1,400	95	—	0.93
S10	K082901	1–3	1,700	97	—	0.94
	K082902	1–3	1,700	97	—	0.95
	K082903	1–3	1,700	97	—	0.96
	K083402	1–4	1,400	92	—	0.94
	K083502	1–3	1,600	89	—	0.85
	K085302	1–3	1,500	92	—	0.88
	K154001	1–3	1,700	95	—	0.93
S11	K076201	1–3	1,600	96	—	0.91
	K080901	1–3	1,400	96	—	0.91
	K089502	1–2	1,700	97	0.93	—
	K106301	1–5	1,600	89	—	0.93
	K106603	1–3	1,500	94	—	0.91

Printer-friendly Version