NAEP Analysis and Scaling → Initial Activities → Constructed-Response Interrater Reliability → Range of Response Codes, Percent Agreement, and Cohen's Kappa or Intraclass Correlation for the Constructed-Response Items Used in Scaling, Grade 12 Writing National Assessment, by Item and Block: 2011

NAEP Technical DocumentationScore range, percent agreement for the constructed-response items used in scaling, grade 12 writing national assessment, by item and block: 2011

Block	Item	Range of response codes	Sample size	Percent exact agreement
NOTE: Special codes assigned to student responses including blank, off-task, and not-scorable were included in the calculation of the Percent Exact Agreement. SOURCE: U.S. Department of Education, Institute of Education Sciences, National Center for Education Statistics, National Assessment of Educational Progress (NAEP), 2011 Writing Assessment.
W1	W019901	1-6	600	60
W2	W021801	1-6	600	67
W3	W020201	1-6	600	60
W4	W022301	1-6	600	60
W5	W022601	1-6	600	65
W6	W019401	1-6	600	61
W7	W019701	1-6	600	57
W8	W020001	1-6	700	65
W9	W020301	1-6	600	59
W10	W021001	1-6	600	62
W11	W021701	1-6	600	62
W12	W021901	1-6	600	62
W13	W022401	1-6	600	60
W14	W022701	1-6	600	63
W15	W019801	1-6	700	58
W16	W020101	1-6	600	65
W17	W020601	1-6	600	58
W18	W020801	1-6	800	61
W19	W020901	1-6	500	64
W20	W022001	1-6	600	68
W21	W022501	1-6	600	65
W22	W022801	1-6	700	62

Score range and Cohen's Kappa or intraclass correlation for the constructed-response items used in scaling, grade 12 writing national assessment, by item and block: 2011

Block	Item	Range of response codes	Sample size	Cohen’s Kappa	Intraclass correlation
NOTE: Cohen's Kappa is a measure of reliability that is appropriate for items that are dichotomously scored. The Intraclass Correlation Coefficient is most appropriate for items with more than two categories. Special codes assigned to student responses including blank, off-task, and not-scorable were not included in the calculation of Cohen's Kappa or intraclass correlation. SOURCE: U.S. Department of Education, Institute of Education Sciences, National Center for Education Statistics, National Assessment of Educational Progress (NAEP), 2011 Writing Assessment.
W1	W019901	1-6	600	0.48	0.81
W2	W021801	1-6	600	0.55	0.86
W3	W020201	1-6	600	0.46	0.80
W4	W022301	1-6	600	0.45	0.80
W5	W022601	1-6	600	0.53	0.85
W6	W019401	1-6	600	0.50	0.84
W7	W019701	1-6	600	0.44	0.82
W8	W020001	1-6	700	0.53	0.85
W9	W020301	1-6	600	0.45	0.78
W10	W021001	1-6	600	0.48	0.78
W11	W021701	1-6	600	0.46	0.78
W12	W021901	1-6	600	0.50	0.83
W13	W022401	1-6	600	0.46	0.82
W14	W022701	1-6	600	0.51	0.83
W15	W019801	1-6	700	0.41	0.74
W16	W020101	1-6	600	0.52	0.81
W17	W020601	1-6	600	0.44	0.81
W18	W020801	1-6	800	0.47	0.80
W19	W020901	1-6	500	0.51	0.83
W20	W022001	1-6	600	0.56	0.84
W21	W022501	1-6	600	0.52	0.84
W22	W022801	1-6	600	0.47	0.81

Printer-friendly Version