Table of Contents  |  Search Technical Documentation  |  References

NAEP Analysis and Scaling → Initial Activities → Constructed-Response Interrater Reliability → Range of Response Codes, Percent Exact Agreement, and Cohen's Kappa or Intraclass Correlation for the Constructed-Response Items From Previous Years, Rescored in 2005, by Block and Item, Grade 8 Science Combined National and State Main Assessment: 2005

NAEP Technical DocumentationRange of response codes, percentage exact agreement, and Cohen's Kappa or intraclass correlation for the constructed-response items from previous years, rescored in 2005, by block and item, grade 8 science combined national and state main assessment: 2005

Block	Item	Range of response codes	Sample size	Percent exact agreement	Cohen's Kappa	Intraclass correlation
B7	K097901	1–3	200	90	†	0.78
	K098501	1–3	200	93	†	0.91
	K101201	1–3	200	92	†	0.55
	K101801	1–3	800	80	†	0.74
	K102001	1–4	200	76	†	0.74
	K102301	1–3	200	97	†	0.97
B8	K041306	1–3	200	78	†	0.81
	K041307	1–3	200	84	†	0.80
	K041401	1–3	200	85	†	0.82
	K041402	1–3	200	93	†	0.89
	K041403	1–3	200	84	†	0.75
B9	K031602	1–3	900	93	†	0.94
	K031603	1–3	900	99	†	0.97
	K031604	1–3	900	98	†	0.96
	K031606	1–3	900	89	†	0.88
	K031607	1–4	900	87	†	0.90
	K031608	1–3	900	87	†	0.80
	K031609	1–3	900	93	†	0.93
	K031610	1–3	900	97	†	0.94
	K031611	1–3	800	98	†	0.94
	K031613	1–3	900	98	†	0.94
B10	K092601	1–3	100	92	†	0.87
	K093601	1–3	500	94	†	0.90
	K095801	1–2	100	95	0.87	†
	K095901	1–3	100	95	†	0.94
	K096101	1–3	100	87	†	0.83
	K099001	1–3	100	88	†	0.78
B11	K043001	1–3	700	92	†	0.80
	K043101	1–3	700	84	†	0.84
	K043102	1–4	600	77	†	0.87
	K043103	1–3	700	92	†	0.79
	K043501	1–3	700	86	†	0.83
	K043601	1–3	600	94	†	0.93
	K043602	1–3	500	97	†	0.88
	K043603	1–3	600	91	†	0.91
B12	K047201	1–4	100	84	†	0.93
	K047301	1–3	100	92	†	0.89
	K047401	1–3	100	91	†	0.91
	K047901	1–3	100	84	†	0.84
	K048001	1–3	100	98	†	0.97
	K048101	1–4	600	91	†	0.88
	K048102	1–3	500	97	†	0.79
	K048103	1–2	600	91	0.80	†
B13	K048601	1–3	100	92	†	0.91
	K048901	1–3	100	98	†	0.98
	K049001	1–3	700	99	†	0.87
	K049301	1–3	100	98	†	0.73
	K049401	1–3	100	88	†	0.70
	K049402	1–3	100	78	†	0.82
	K049403	1–3	100	91	†	0.79
	K049404	1–4	100	84	†	0.81
B14	K035601	1–3	200	84	†	0.83
	K035801	1–3	200	90	†	0.91
	K035901	1–3	200	92	†	0.88
	K036101	1–3	200	83	†	0.79
	K036301	1–3	200	91	†	0.94
	K036401	1–3	200	94	†	0.91
	K036402	1–3	200	90	†	0.86
	K036403	1–3	200	89	†	0.81
	K036404	1–3	200	88	†	0.47
	K036701	1–3	800	97	†	0.92
	K036801	1–3	200	93	†	0.88
B15	K037301	1–3	100	99	†	0.99
	K037401	1–3	100	84	†	0.59
	K037501	1–3	600	96	†	0.91
	K037601	1–3	100	88	†	0.90
	K037701	1–4	100	100	†	1.00
	K037703	1–3	100	92	†	0.87
	K038101	1–2	100	99	0.96	†
	K038201	1–3	100	94	†	0.82
	K038301	1–5	100	89	†	0.91
B20	K092801	1–3	200	92	†	0.87
	K093701	1–3	100	91	†	0.88
	K093901	1–3	200	85	†	0.87
	K094901	1–3	100	87	†	0.58
	K095401	1–3	100	93	†	0.82
	K097001	1–3	100	91	†	0.85
B21	K045301	1–3	700	84	†	0.83
	K045601	1–4	700	96	†	0.94
	K045701	1–3	700	81	†	0.83
	K045801	1–3	700	89	†	0.85
	K046301	1–3	700	95	†	0.93
	K046401	1–3	600	94	†	0.87
	K046501	1–3	600	87	†	0.83
	K046601	1–3	600	84	†	0.82
	K046701	1–4	600	88	†	0.88
† The intraclass correlation is not reported for dichotomously scored items; Cohen's Kappa is not reported for polytomously scored items. NOTE: Cohen's Kappa is a measure of reliability that is appropriate for items that are dichotomously scored. The intraclass correlation coefficient is most appropriate for items with more than two categories. SOURCE: U.S. Department of Education, Institute of Education Sciences, National Center for Education Statistics, National Assessment of Educational Progress (NAEP), 2005 Science Assessment.

Printer-friendly Version