Table of Contents  |  Search Technical Documentation  |  References

NAEP Analysis and Scaling → Initial Activities → Constructed-Response Interrater Reliability → Range of Response Codes, Percent Agreement, and Cohen's Kappa or Intraclass Correlation for the Constructed-Response Items Used in Scaling, Grade 8 Writing Combined National and State Main Assessment, by Item and Block: 2007

NAEP Technical DocumentationRange of response codes, percent agreement, and Cohen's Kappa or intraclass correlation for the constructed-response items used in scaling, grade 8 writing combined national and state main assessment, by item and block: 2007

Block	Item	Range of response codes	Sample size	Percent agreement	Cohen’s Kappa	Intraclass correlation
— Not available. The intraclass correlation is not reported for dichotomously scored items; Cohen's Kappa is not reported for polytomously scored items. NOTE: Cohen's Kappa is a measure of reliability that is appropriate for items that are dichotomously scored. The intraclass correlation coefficient is most appropriate for items with more than two categories. SOURCE: U.S. Department of Education, Institute of Education Sciences, National Center for Education Statistics, National Assessment of Educational Progress (NAEP), 2007 Writing Assessment.
W3	W006002	1–6	700	77.94	—	0.88
W4	W011502	1–6	700	68.09	—	0.84
W5	W006202	1–6	700	69.01	—	0.87
W6	W006302	1–6	700	64.29	—	0.79
W7	W006402	1–6	700	84.38	—	0.93
W8	W006502	1–6	700	74.30	—	0.88
W9	W010902	1–6	700	73.31	—	0.79
W11	W006802	1–6	700	67.64	—	0.85
W12	W006902	1–6	700	74.68	—	0.84
W13	W007002	1–6	700	68.72	—	0.81
W14	W007102	1–6	700	72.53	—	0.84
W15	W011002	1–6	700	68.79	—	0.84
W16	W011602	1–6	700	74.78	—	0.83
W17	W007402	1–6	700	67.40	—	0.86
W19	W011102	1–6	700	71.89	—	0.84
W20	W011702	1–6	700	67.69	—	0.82
W21	W007802	1–6	700	66.19	—	0.82
W22	W007902	1–6	700	71.51	—	0.85
W23	W008002	1–6	700	77.86	—	0.87
W24	W008102	1–6	700	70.91	—	0.86

Printer-friendly Version