Table of Contents  |  Search Technical Documentation  |  References

NAEP Analysis and Scaling → Initial Activities → Constructed-Response Interrater Reliability → Range of Response Codes, Percentage Exact Agreement, and Cohen's Kappa or Intraclass Correlation for the Constructed-Response Items From 1996 That Were Rescored in 2000, by Block and Item, Grade 12 Science Assessment: 2000

NAEP Technical DocumentationRange of response codes, percentage exact agreement, and Cohen's Kappa or intraclass correlation for the constructed-response items from 1996 that were rescored in 2000, by block and item, grade 12 science assessment: 2000

Block	Item	Range of response codes	Sample size	Percentage exact agreement	Cohen's Kappa	Intraclass correlation
S3	K049501	1–4	600	98	†	0.91
	K049502	1–5	600	91	†	0.91
	K049503	1–3	600	91	†	0.93
	K049504	1–3	600	87	†	0.88
	K049505	1–3	600	92	†	0.92
	K049506	1–3	600	91	†	0.79
S5	K040801	1–3	1,100	98	†	0.93
	K040802	1–3	1,100	99	†	0.94
	K040803	1–3	1,100	97	†	0.96
	K040804	1–3	800	85	†	0.85
	K040805	1–3	800	92	†	0.89
	K040806	1–2	700	83	0.64	†
	K040808	1–3	1,000	98	†	0.98
	K040809	1–3	1,000	95	†	0.93
S6	K049701	1–3	400	97	†	0.93
	K049702	1–3	400	98	†	0.96
	K049703	1–3	400	92	†	0.91
	K049704	1–3	400	92	†	0.88
	K049705	1–4	400	75	†	0.81
	K049706	1–3	400	73	†	0.75
	K049707	1–5	400	78	†	0.88
	K049708	1–2	400	94	0.86	†
S8	K041306	1–3	600	84	†	0.87
	K041307	1–3	600	82	†	0.81
	K041401	1–3	600	92	†	0.89
	K041402	1–3	600	93	†	0.89
	K041403	1–3	600	91	†	0.83
	K041404	1–3	600	93	†	0.93
	K041406	1–3	600	94	†	0.80
S9	K049901	1–3	600	97	†	0.97
	K049902	1–3	600	81	†	0.80
	K049903	1–3	600	89	†	0.86
	K049904	1–3	600	98	†	0.98
	K049907	1–3	600	95	†	0.95
	K049908	1–3	600	95	†	0.95
	K049909	1–4	600	96	†	0.95
	K049911	1–3	600	92	†	0.92
	K049912	1–5	600	74	†	0.85
	K049914	1–3	600	97	†	0.98
S11	K051701	1–2	600	93	0.85	†
	K051801	1–3	600	95	†	0.96
	K052301	1–3	600	99	†	0.99
	K052401	1–4	600	87	†	0.92
	K052402	1–3	600	92	†	0.84
	K052501	1–4	600	86	†	0.91
	K052502	1–4	600	89	†	0.89
	K052503	1–3	600	90	†	0.77
S12	K047201	1–4	600	93	†	0.97
	K047301	1–3	600	96	†	0.97
	K047401	1–3	600	85	†	0.90
	K047901	1–3	600	90	†	0.90
	K048001	1–3	600	98	†	0.94
	K048101	1–4	500	83	†	0.84
	K048102	1–3	500	90	†	0.64
	K048103	1–2	600	88	0.77	†
S13	K048601	1–3	600	94	†	0.94
	K048901	1–3	600	98	†	0.98
	K049001	1–3	600	98	†	0.83
	K049301	1–3	600	97	†	0.92
	K049401	1–3	600	91	†	0.88
	K049402	1–3	600	83	†	0.83
	K049403	1–3	600	73	†	0.68
	K049404	1–4	600	65	†	0.63
S14	K052901	1–4	600	93	†	0.93
	K053001	1–3	600	90	†	0.81
	K053101	1–3	600	99	†	0.98
	K053102	1–3	600	97	†	0.97
	K053601	1–5	600	92	†	0.92
	K053701	1–3	600	91	†	0.83
	K053801	1–3	600	87	†	0.86
	K053901	1–3	600	91	†	0.79
S15	K054001	1–4	600	94	†	0.95
	K054002	1–3	600	93	†	0.89
	K054003	1–3	600	100	†	0.96
	K054004	1–2	600	100	0.97	†
	K054005	1–3	600	90	†	0.82
	K054006	1–3	600	93	†	0.85
	K054007	1–3	600	89	†	0.87
	K054008	1–4	600	94	†	0.95
S21	K059001	1–3	600	82	†	0.73
	K059101	1–3	600	93	†	0.85
	K059201	1–4	600	92	†	0.94
	K059301	1–4	600	99	†	0.96
	K059801	1–3	600	89	†	0.80
	K059901	1–4	600	96	†	0.96
	K060001	1–3	600	86	†	0.89
	K060101	1–4	600	68	†	0.65
† The intraclass correlation is not reported for dichotomously scored items; Cohen's Kappa is not reported for polytomously scored items. NOTE: Cohen's Kappa is a measure of reliability that is appropriate for items that are dichotomously scored. The intraclass correlation coefficient is most appropriate for items with more than two categories. SOURCE: U.S. Department of Education, Institute of Education Sciences, National Center for Education Statistics, National Assessment of Educational Progress (NAEP), 2000 Science Assessment.

Printer-friendly Version