Table of Contents  |  Search Technical Documentation  |  References

NAEP Analysis and Scaling → Initial Activities → Constructed-Response Interrater Reliability → Range of Response Codes, Percent Exact Agreement, and Cohen's Kappa or Intraclass Correlation for the Constructed-Response Items From Previous Years, Rescored in 2005, by Block and Item, Grade 4 Reading Combined National and State Main Assessment: 2005

NAEP Technical DocumentationRange of response codes, percentage exact agreement, and Cohen's Kappa or intraclass correlation for the constructed-response items from previous years, rescored in 2005, by block and item, grade 4 reading combined national and state main assessment: 2005

Block	Item	Range of response codes	Sample size	Percent exact agreement	Cohen's Kappa	Intraclass correlation
R3	R017001	1–2	600	90	0.75	†
	R017003	1–3	2,700	82	†	0.82
	R017004	1–2	1,100	94	0.83	†
	R017006	1–2	1,200	90	0.75	†
	R017007	1–4	4,100	71	†	0.84
	R017009	1–3	600	82	†	0.71
R4	R052904	1–3	1,500	86	†	0.89
	R052906	1–2	1,400	91	0.82	†
	R052907	1–4	4,300	80	†	0.87
	R052910	1–4	700	92	†	0.79
R5	R012601	1–2	600	88	0.72	†
	R012604	1–2	1,300	93	0.80	†
	R012607	1–4	3,300	81	†	0.85
	R012612	1–2	4,200	83	0.64	†
R6	R017301	1–2	1,800	86	0.68	†
	R017303	1–3	1,800	86	†	0.89
	R017307	1–4	3,000	71	†	0.78
	R017309	1–3	1,800	83	†	0.84
	R017310	1–3	1,600	89	†	0.91
R7	R012702	1–2	2,000	93	0.83	†
	R012703	1–2	1,400	89	0.73	†
	R012705	1–2	2,200	89	0.66	†
	R012706	1–2	2,200	81	0.54	†
	R012710	1–2	400	90	0.76	†
	R012714	1–4	4,000	76	†	0.84
R8	R017401	1–3	1,700	76	†	0.76
	R017701	1–3	3,300	74	†	0.73
	R017901	1–4	2,200	75	†	0.84
	R018201	1–3	1,300	77	†	0.76
	R018301	1–3	1,900	79	†	0.76
R9	R053003	1–2	500	96	0.92	†
	R053004	1–3	2,200	87	†	0.88
	R053006	1–4	1,800	76	†	0.89
	R053009	1–2	1,500	91	0.77	†
	R053010	1–3	1,300	85	†	0.84
R10	R053101	1–2	1,200	89	0.71	†
	R053105	1–3	2,000	82	†	0.83
	R053106	1–4	1,600	85	†	0.93
	R053108	1–3	1,500	88	†	0.87
R11	R020401	1–3	1,200	88	†	0.85
	R020701	1–2	1,300	91	0.76	†
	R021201	1–3	2,700	85	†	0.86
	R021701	1–3	3,500	75	†	0.73
R12	R023201	1–2	600	89	0.77	†
	R023501	1–3	3,100	78	†	0.83
	R023601	1–3	3,400	81	†	0.78
	R024001	1–3	3,500	78	†	0.77
† The intraclass correlation is not reported for dichotomously scored items; Cohen's Kappa is not reported for polytomously scored items. NOTE: Cohen's Kappa is a measure of reliability that is appropriate for items that are dichotomously scored. The intraclass correlation coefficient is most appropriate for items with more than two categories. SOURCE: U.S. Department of Education, Institute of Education Sciences, National Center for Education Statistics, National Assessment of Educational Progress (NAEP), 2005 Reading Assessment.

Printer-friendly Version