NAEP Analysis and Scaling → Initial Activities → Constructed-Response Interrater Reliability → Range of Response Codes, Percent Agreement, and Cohen's Kappa or Intraclass Correlation for the Constructed-Response Items From Previous Years, Rescored in 2009, Grade 8 Reading Combined National and State Assessments, by Item and Block: 2011

NAEP Technical DocumentationScore range and percent agreement for the constructed-response items from the previous year assessed that were rescored in 2009, grade 8 reading combined national and state assessments, by item and block: 2011

Block	Item	Range of response codes	Sample size	Percent exact agreement
NOTE: Special codes assigned to student responses including blank, off-task, and not-scorable were included in the calculation of the Percent Exact Agreement measure. SOURCE: U.S. Department of Education, Institute of Education Sciences, National Center for Education Statistics, National Assessment of Educational Progress (NAEP), 2011 Reading Assessment.
R1	R059604	1-2	1,700	91
	R059605	1-3	1,600	80
	R059606	1-3	1,600	84
	R059607	1-3	1,600	88
	R059610	1-4	1,600	85
R2	R059803	1-4	1,600	76
	R059806	1-3	1,600	85
	R059807	1-3	1,600	89
	R059809	1-2	1,600	91
R3	R060103	1-2	1,600	85
	R060105	1-2	1,600	92
	R060108	1-4	1,600	72
R4	R060301	1-2	1,600	91
	R060305	1-3	1,600	84
	R060306	1-4	1,600	79
	R060309	1-3	1,600	67
R5	R058504	1-3	1,600	75
	R058508	1-4	1,600	70
	R058509	1-2	1,600	90
R6	R058602	1-3	1,600	78
	R058606	1-3	1,600	82
	R058608	1-4	1,600	74
R7	R058805	1-3	1,600	83
	R058807	1-4	1,600	72
	R058810	1-3	1,600	81
R8	R059106	1-4	1,600	69
	R059108	1-3	1,700	75
	R059109	1-3	1,600	77
	R059110	1-2	1,600	91
R9	R060503	1-3	1,600	90
	R060505	1-4	1,600	79
	R060507	1-3	1,600	80
	R060508	1-3	1,600	74
	R060509	1-3	1,600	82
R11	R060803	1-3	1,600	70
	R060807	1-2	1,600	94
	R060808	1-4	1,600	76
	R060810	1-3	1,600	75
	R060811	1-3	1,600	81
R13	R061301	1-3	1,600	88
	R061303	1-3	1,600	77
	R061304	1-2	1,600	93
	R061307	1-4	1,600	74
	R061309	1-3	1,600	74

Score range and Cohen's Kappa or intraclass correlation for the constructed-response items from the previous year assessed that were rescored in 2009, grade 8 reading combined national and state assessments, by item and block: 2011

Block	Item	Range of response codes	Sample size	Cohen’s Kappa	Intraclass correlation
† The intraclass correlation is not reported for dichotomously scored items. NOTE: Cohen's Kappa is a measure of reliability that is appropriate for items that are dichotomously scored. The Intraclass Correlation Coefficient is most appropriate for items with more than two categories. Special codes assigned to student responses including blank, off-task, and not-scorable were not included in the calculation of Cohen's Kappa or intraclass correlation. SOURCE: U.S. Department of Education, Institute of Education Sciences, National Center for Education Statistics, National Assessment of Educational Progress (NAEP), 2011 Reading Assessment.
R1	R059604	1-2	1,500	0.69	†
	R059605	1-3	1,500	0.63	0.72
	R059606	1-3	1,500	0.72	0.81
	R059607	1-3	1,400	0.78	0.88
	R059610	1-4	1,500	0.77	0.90
R2	R059803	1-4	1,500	0.63	0.79
	R059806	1-3	1,400	0.67	0.76
	R059807	1-3	1,500	0.78	0.84
	R059809	1-2	1,500	0.80	†
R3	R060103	1-2	1,400	0.65	†
	R060105	1-2	1,500	0.84	†
	R060108	1-4	1,500	0.59	0.74
R4	R060301	1-2	1,500	0.72	†
	R060305	1-3	1,400	0.67	0.75
	R060306	1-4	1,500	0.69	0.82
	R060309	1-3	1,400	0.41	0.56
R5	R058504	1-3	1,500	0.53	0.63
	R058508	1-4	1,500	0.51	0.77
	R058509	1-2	1,500	0.78	†
R6	R058602	1-3	1,400	0.61	0.66
	R058606	1-3	1,500	0.68	0.72
	R058608	1-4	1,500	0.60	0.80
R7	R058805	1-3	1,500	0.70	0.79
	R058807	1-4	1,500	0.53	0.78
	R058810	1-3	1,500	0.64	0.71
R8	R059106	1-4	1,400	0.49	0.72
	R059108	1-3	1,500	0.58	0.78
	R059109	1-3	1,500	0.61	0.72
	R059110	1-2	1,500	0.78	†
R9	R060503	1-3	1,500	0.84	0.90
	R060505	1-4	1,500	0.70	0.86
	R060507	1-3	1,500	0.66	0.61
	R060508	1-3	1,500	0.50	0.60
	R060509	1-3	1,400	0.69	0.78
R11	R060803	1-3	1,500	0.46	0.58
	R060807	1-2	1,400	0.86	†
	R060808	1-4	1,400	0.63	0.83
	R060810	1-3	1,400	0.57	0.71
	R060811	1-3	1,500	0.67	0.76
R13	R061301	1-3	1,500	0.79	0.85
	R061303	1-3	1,500	0.51	0.62
	R061304	1-2	1,500	0.85	†
	R061307	1-4	1,400	0.61	0.83
	R061309	1-3	1,400	0.53	0.71

Printer-friendly Version