Table of Contents  |  Search Technical Documentation  |  References

NAEP Analysis and Scaling → Initial Activities → Constructed-Response Interrater Reliability → Range of Response Codes, Percent Exact Agreement, and Cohen's Kappa or Intraclass Correlation for the Constructed-Response Items Used in Scaling, by Block and Item, Grade 4 Reading Combined National and State Main Assessment: 2005

NAEP Technical DocumentationRange of response codes, percentage exact agreement, and Cohen's Kappa or intraclass correlation for the constructed-response items used in scaling, by block and item, grade 4 reading combined national and state main assessment: 2005

Block	Item	Range of response codes	Sample size	Percent exact agreement	Cohen's Kappa	Intraclass correlation
R3	R017001	1–2	1,700	90	0.76	†
	R017003	1–3	1,600	91	†	0.92
	R017004	1–2	1,600	95	0.87	†
	R017006	1–2	1,500	91	0.79	†
	R017007	1–4	1,400	80	†	0.90
	R017009	1–3	1,300	83	†	0.71
R4	R052904	1–3	1,600	90	†	0.90
	R052906	1–2	1,500	95	0.90	†
	R052907	1–4	1,400	82	†	0.85
	R052909	1–2	1,300	94	0.84	†
R5	R012601	1–2	1,600	94	0.87	†
	R012604	1–2	1,500	96	0.89	†
	R012607	1–4	1,400	80	†	0.80
	R012612	1–2	1,300	91	0.81	†
R6	R017301	1–2	1,600	89	0.72	†
	R017303	1–3	1,500	89	†	0.91
	R017307	1–4	1,300	84	†	0.84
	R017309	1–3	1,100	87	†	0.87
	R017310	1–3	1,500	97	†	0.98
R7	R012702	1–2	1,600	94	0.84	†
	R012703	1–2	1,600	90	0.77	†
	R012705	1–2	1,400	93	0.71	†
	R012706	1–2	1,500	89	0.71	†
	R012710	1–2	1,100	91	0.81	†
	R012714	1–4	1,300	80	†	0.80
R8	R017401	1–3	1,600	79	†	0.77
	R017701	1–3	1,500	77	†	0.75
	R017901	1–4	1,300	80	†	0.82
	R018201	1–3	1,300	77	†	0.77
	R018301	1–3	1,200	81	†	0.77
R9	R053003	1–2	1,700	97	0.92	†
	R053004	1–3	1,700	93	†	0.94
	R053006	1–4	1,500	82	†	0.93
	R053009	1–2	1,500	91	0.73	†
	R053010	1–3	1,400	88	†	0.86
R10	R053101	1–2	1,600	92	0.75	†
	R053105	1–3	1,600	85	†	0.86
	R053106	1–4	1,400	89	†	0.96
	R053108	1–3	1,400	93	†	0.91
R11	R020401	1–3	1,700	91	†	0.88
	R020701	1–2	1,600	94	0.82	†
	R021201	1–3	1,500	80	†	0.80
	R021701	1–3	1,600	74	†	0.65
R12	R023201	1–2	1,400	88	0.76	†
	R023501	1–3	1,500	88	†	0.91
	R023601	1–3	1,400	87	†	0.86
	R024001	1–3	1,400	88	†	0.86
† The intraclass correlation is not reported for dichotomously scored items; Cohen's Kappa is not reported for polytomously scored items. NOTE: Cohen's Kappa is a measure of reliability that is appropriate for items that are dichotomously scored. The intraclass correlation coefficient is most appropriate for items with more than two categories. SOURCE: U.S. Department of Education, Institute of Education Sciences, National Center for Education Statistics, National Assessment of Educational Progress (NAEP), 2005 Reading Assessment.

Printer-friendly Version