NAEP Analysis and Scaling → Initial Activities → Constructed-Response Interrater Reliability → Range of Response Codes, Percent Agreement, and Cohen's Kappa or Intraclass Correlation for the Constructed-Response Items Used in Scaling, Grade 8 Writing National Assessment, by Item and Block: 2011

NAEP Technical DocumentationScore range and percent agreement for the constructed-response items used in scaling, grade 8 writing national assessment, by item and block: 2011

Block	Item	Range of response codes	Sample size	Percent exact agreement
NOTE: Special codes assigned to student responses including blank, off-task, and not-scorable were included in the calculation of the Percent Exact Agreement. SOURCE: U.S. Department of Education, Institute of Education Sciences, National Center for Education Statistics, National Assessment of Educational Progress (NAEP), 2011 Writing Assessment.
W1	W015801	1-6	500	61
W2	W015901	1-6	600	61
W3	W016101	1-6	500	59
W4	W018801	1-6	600	60
W5	W018901	1-6	500	60
W6	W019001	1-6	600	59
W7	W016901	1-6	600	65
W8	W017101	1-6	600	57
W9	W017201	1-6	600	59
W10	W017401	1-6	600	61
W11	W018301	1-6	500	59
W12	W018501	1-6	600	62
W13	W018601	1-6	500	62
W14	W018701	1-6	600	62
W15	W016401	1-6	600	67
W16	W016501	1-6	600	61
W17	W016601	1-6	600	58
W18	W016801	1-6	600	62
W19	W017501	1-6	500	57
W20	W017601	1-6	600	59
W21	W017801	1-6	500	58
W22	W018001	1-6	600	60

Score range and Cohen's Kappa or intraclass correlation for the constructed-response items used in scaling, grade 8 writing national assessment, by item and block: 2011

Block	Item	Range of response codes	Sample size	Cohen’s Kappa	Intraclass correlation
NOTE: Cohen's Kappa is a measure of reliability that is appropriate for items that are dichotomously scored. The Intraclass Correlation Coefficient is most appropriate for items with more than two categories. Special codes assigned to student responses including blank, off-task, and not-scorable were not included in the calculation of Cohen's Kappa or intraclass correlation. SOURCE: U.S. Department of Education, Institute of Education Sciences, National Center for Education Statistics, National Assessment of Educational Progress (NAEP), 2011 Writing Assessment.
W1	W015801	1-6	500	0.48	0.79
W2	W015901	1-6	500	0.47	0.80
W3	W016101	1-6	500	0.44	0.79
W4	W018801	1-6	600	0.47	0.80
W5	W018901	1-6	500	0.45	0.81
W6	W019001	1-6	600	0.47	0.82
W7	W016901	1-6	600	0.54	0.86
W8	W017101	1-6	500	0.44	0.79
W9	W017201	1-6	600	0.43	0.72
W10	W017401	1-6	600	0.45	0.77
W11	W018301	1-6	500	0.44	0.79
W12	W018501	1-6	600	0.47	0.76
W13	W018601	1-6	500	0.48	0.82
W14	W018701	1-6	600	0.48	0.80
W15	W016401	1-6	500	0.56	0.85
W16	W016501	1-6	600	0.49	0.82
W17	W016601	1-6	600	0.44	0.83
W18	W016801	1-6	500	0.51	0.85
W19	W017501	1-6	500	0.46	0.83
W20	W017601	1-6	600	0.48	0.82
W21	W017801	1-6	500	0.44	0.80
W22	W018001	1-6	600	0.45	0.79

Printer-friendly Version