Table of Contents  |  Search Technical Documentation  |  References

NAEP Analysis and Scaling → Initial Activities → Constructed-Response Interrater Reliability → Range of Response Codes, Percentage Exact Agreement, and Cohen's Kappa or Intraclass Correlation for the Constructed-Response Items From Previous Assessment Years, Rescored in 2003, by Block and Item, Grade 8 Reading Combined National and State Main Assessment: 2003

NAEP Technical DocumentationRange of response codes, percentage exact agreement, and Cohen's Kappa or intraclass correlation for the constructed-response items from previous assessment years, rescored in 2003, by block and item, grade 8 reading combined national and state main assessment: 2003

Block	Item	Range of response codes	Sample size	Percent exact agreement	Cohen's Kappa	Intraclass correlation
R3	R017101	1–2	1,000	93	0.85	†
	R017102	1–3	800	86	†	0.87
	R017104	1–3	1,200	92	†	0.93
	R017105	1–4	3,800	76	†	0.86
	R017107	1–3	1,200	89	†	0.90
	R017108	1–2	600	93	0.87	†
	R017110	1–3	800	92	†	0.94
R4	R018401	1–3	1,800	80	†	0.85
	R018601	1–3	2,200	79	†	0.80
	R018701	1–3	1,200	82	†	0.83
	R018801	1–3	1,600	89	†	0.88
	R019001	1–4	3,600	86	†	0.94
	R019101	1–3	800	88	†	0.90
	R019201	1–3	1,000	85	†	0.85
R5	R012601	1–2	1,000	84	0.68	†
	R012604	1–2	2,200	90	0.80	†
	R012607	1–4	4,800	77	†	0.83
	R012612	1–2	3,000	82	0.61	†
R6	R013201	1–4	2,200	80	†	0.86
	R013203	1–2	600	93	0.66	†
	R013205	1–2	600	92	0.74	†
	R013207	1–2	1,400	90	0.78	†
	R013209	1–2	700	96	0.84	†
	R013211	1–2	800	85	0.68	†
	R013212	1–4	2,600	88	†	0.84
R7	R012702	1–2	800	92	0.66	†
	R012703	1–2	800	84	0.67	†
	R012705	1–2	2,900	86	0.72	†
	R012706	1–2	800	83	0.64	†
	R012710	1–2	700	85	0.66	†
	R012713	1–2	600	98	0.95	†
	R012714	1–4	5,500	74	†	0.84
R8	R017202	1–3	1,000	87	†	0.89
	R017204	1–3	1,200	76	†	0.76
	R017205	1–4	3,200	64	†	0.78
	R017207	1–3	800	83	†	0.81
	R017208	1–3	1,600	82	†	0.82
	R017210	1–2	1,400	91	0.82	†
R9	R016101	1–3	1,400	79	†	0.76
	R016107	1–3	600	94	†	0.95
	R016108	1–3	700	70	†	0.72
	R016109	1–3	700	88	†	0.83
R10	R013402	1–2	600	97	0.95	†
	R013403	1–4	800	97	†	0.97
	R013405	1–2	1,200	91	0.81	†
	R013406	1–4	3,600	75	†	0.88
	R013407	1–2	900	91	0.80	†
	R013409	1–2	700	91	0.82	†
	R013411	1–2	600	94	0.85	†
	R013413	1–2	1,200	85	0.71	†
R11	R013001	1–2	600	91	0.74	†
	R013003	1–2	700	100	1.00	†
	R013004	1–4	3,800	68	†	0.82
	R013005	1–2	800	90	0.74	†
	R013007	1–2	200	95	0.83	†
	R013008	1–2	600	88	0.75	†
	R013009	1–2	700	91	0.65	†
	R013010	1–2	600	95	0.81	†
	R013011	1–2	600	81	0.60	†
R12	R024401	1–3	1,400	86	†	0.83
	R024501	1–3	1,800	79	†	0.75
	R024801	1–4	2,200	77	†	0.79
	R025001	1–3	200	89	†	0.89
	R025601	1–3	1,000	82	†	0.75
	R026101	1–2	800	88	0.77	†
R13	R016201	1–3	1,400	92	†	0.62
	R016202	1–3	1,200	85	†	0.73
	R016204	1–4	1,400	90	†	0.81
	R016205	1–3	1,200	83	†	0.75
	R016207	1–3	800	91	†	0.92
	R016210	1–4	4,400	71	†	0.72
	R016211	1–3	1,600	89	†	0.81
	R016212	1–3	700	91	†	0.92
	R016213	1–3	1,200	86	†	0.81
R14	R026501	1–3	2,400	85	†	0.86
	R026801	1–4	2,000	80	†	0.89
	R027001	1–2	1,000	89	0.78	†
	R027201	1–3	1,600	87	†	0.88
R15	R028401	1–3	800	89	†	0.90
	R028501	1–3	600	89	†	0.91
	R028801	1–4	2,800	82	†	0.88
	R029601	1–3	1,400	85	†	0.78
† The intraclass correlation is not reported for dichotomously scored items; Cohen's Kappa is not reported for polytomously scored items. NOTE: Cohen's Kappa is a measure of reliability that is appropriate for items that are dichotomously scored. The intraclass correlation coefficient is most appropriate for items with more than two categories. SOURCE: U.S. Department of Education, Institute of Education Sciences, National Center for Education Statistics, National Assessment of Educational Progress (NAEP), 2003 Reading Assessment.

Printer-friendly Version