Table of Contents  |  Search Technical Documentation  |  References

NAEP Analysis and Scaling → Initial Activities → Constructed-Response Interrater Reliability → Range of Response Codes, Percent Exact Agreement, and Cohen's Kappa or Intraclass Correlation for the Constructed-Response Items Used in Scaling, by Block and Item, Grade 8 Science Combined National and State Main Assessment: 2005

NAEP Technical DocumentationRange of response codes, percentage exact agreement, and Cohen's Kappa or intraclass correlation for the constructed-response items used in scaling, by block and item, grade 8 science combined national and state main assessment: 2005

Block	Item	Range of response codes	Sample size	Percent exact agreement	Cohen's Kappa	Intraclass correlation
B3	K040601	1–3	900	98	†	0.97
	K040603	1–3	900	94	†	0.93
	K040604	1–3	900	88	†	0.85
	K040605	1–3	800	89	†	0.89
	K040606	1–4	800	90	†	0.91
	K040607	1–4	1,000	91	†	0.95
	K040608	1–4	1,000	89	†	0.95
	K040609	1–4	1,000	96	†	0.98
	K040610	1–4	1,000	93	†	0.96
B5	K040801	1–3	1,000	97	†	0.95
	K040802	1–3	1,000	99	†	0.93
	K040803	1–3	800	96	†	0.95
	K040804	1–3	700	93	†	0.91
	K040805	1–3	600	94	†	0.90
	K040806	1–2	600	87	0.60	†
	K040808	1–3	800	97	†	0.98
	K040809	1–3	800	93	†	0.95
B6	K031301	1–4	1,100	97	†	0.82
	K031302	1–3	1,100	87	†	0.73
	K031305	1–3	1,000	92	†	0.74
	K031306	1–2	1,000	97	0.70	†
	K031307	1–3	900	92	†	0.85
	K031308	1–3	900	95	†	0.72
	K031309	1–4	1,100	97	†	0.89
B7	K097901	1–3	1,200	92	†	0.87
	K098501	1–3	1,400	93	†	0.93
	K101201	1–3	1,200	86	†	0.59
	K102001	1–4	1,300	81	†	0.82
	K102301	1–3	1,400	96	†	0.96
B8	K041306	1–3	1,500	85	†	0.88
	K041307	1–3	1,500	85	†	0.80
	K041401	1–3	1,400	88	†	0.86
	K041402	1–3	1,400	93	†	0.87
	K041403	1–3	1,400	91	†	0.84
B10	K092601	1–3	900	89	†	0.84
	K095801	1–2	900	92	0.78	†
	K095901	1–3	1,000	94	†	0.92
	K096101	1–3	1,000	92	†	0.87
	K099001	1–3	1,000	92	†	0.88
B12	K047201	1–4	1,000	87	†	0.93
	K047301	1–3	900	91	†	0.89
	K047401	1–3	1,000	94	†	0.93
	K047901	1–3	900	88	†	0.90
	K048001	1–3	1,000	95	†	0.89
B13	K048601	1–3	1,000	95	†	0.94
	K048901	1–3	1,000	95	†	0.96
	K049301	1–3	1,000	99	†	0.96
	K049401	1–3	900	90	†	0.81
	K049402	1–3	900	84	†	0.85
	K049403	1–3	900	85	†	0.76
	K049404	1–4	900	78	†	0.67
B14	K035601	1–3	900	90	†	0.86
	K035801	1–3	1,000	91	†	0.93
	K035901	1–3	900	92	†	0.90
	K036101	1–3	900	87	†	0.84
	K036301	1–3	900	93	†	0.94
	K036401	1–3	1,000	94	†	0.91
	K036402	1–3	900	88	†	0.84
	K036403	1–3	900	90	†	0.82
	K036404	1–3	900	89	†	0.70
	K036801	1–3	800	94	†	0.93
B15	K037301	1–3	900	95	†	0.95
	K037401	1–3	700	89	†	0.80
	K037601	1–3	900	86	†	0.86
	K037701	1–4	1,000	99	†	0.98
	K037703	1–3	900	88	†	0.85
	K038101	1–2	900	97	0.87	†
	K038201	1–3	900	91	†	0.74
	K038301	1–5	800	86	†	0.90
B20	K092801	1–3	1,000	96	†	0.94
	K093701	1–3	800	95	†	0.93
	K093901	1–3	900	88	†	0.88
	K094901	1–3	900	91	†	0.78
	K095401	1–3	900	94	†	0.84
	K097001	1–3	900	89	†	0.82
S9	K060301	1–3	1,400	94	†	0.92
	K063403	1–3	1,400	87	†	0.81
	K063404	1–3	1,500	93	†	0.91
	K063405	1–4	1,500	99	†	0.99
	K063406	1–3	1,500	85	†	0.84
	K063407	1–3	1,500	90	†	0.91
	K063410	1–4	1,500	86	†	0.91
S15	K063901	1–3	1,000	98	†	0.97
	K064101	1–4	900	97	†	0.96
	K064501	1–4	900	82	†	0.87
	K064801	1–3	900	93	†	0.89
	K064802	1–3	900	93	†	0.93
	K064803	1–3	900	94	†	0.94
S16	K065601	1–4	900	93	†	0.96
	K065901	1–4	900	84	†	0.85
	K065902	1–3	900	93	†	0.92
	K066101	1–3	900	96	†	0.96
S17	K066701	1–3	900	88	†	0.86
	K066901	1–2	900	99	0.95	†
	K067201	1–3	1,000	94	†	0.94
	K067401	1–3	900	89	†	0.90
	K067701	1–4	900	96	†	0.98
	K067901	1–3	900	92	†	0.92
S18	K063001	1–3	800	90	†	0.92
	K068801	1–3	900	82	†	0.82
	K069101	1–3	1,000	94	†	0.93
	K069501	1–3	900	90	†	0.76
	K069801	1–4	900	90	†	0.88
	K069901	1–4	900	96	†	0.97
† The intraclass correlation is not reported for dichotomously scored items; Cohen's Kappa is not reported for polytomously scored items. NOTE: Cohen's Kappa is a measure of reliability that is appropriate for items that are dichotomously scored. The intraclass correlation coefficient is most appropriate for items with more than two categories. SOURCE: U.S. Department of Education, Institute of Education Sciences, National Center for Education Statistics, National Assessment of Educational Progress (NAEP), 2005 Science Assessment.

Printer-friendly Version