NAEP Analysis and Scaling → Initial Activities → Constructed-Response Interrater Reliability → Range of Response Codes, Percent Agreement, and Cohen's Kappa or Intraclass Correlation for the Constructed-Response Items From Previous Years, Rescored in 2011, Grade 4 Reading Combined National and State Assessments, by Item and Block: 2011

NAEP Technical DocumentationScore range, percent agreement for the constructed-response items from the previous year assessed that were rescored in 2009, grade 4 reading combined national and state assessments, by item and block: 2011

Block	Item	Range of response codes	Sample size	Percent exact agreement
NOTE: Special codes assigned to student responses including blank, off-task, and not-scorable were included in the calculation of the Percent Exact Agreement measure. SOURCE: U.S. Department of Education, Institute of Education Sciences, National Center for Education Statistics, National Assessment of Educational Progress (NAEP), 2011 Reading Assessment.
R1	R057704	1-2	1,600	89
	R057705	1-3	1,600	85
	R057706	1-3	1,600	84
	R057707	1-4	1,600	78
	R057710	1-3	1,600	79
R2	R057806	1-4	1,600	80
	R057807	1-2	1,600	94
	R057809	1-3	1,600	84
R4	R058204	1-3	1,600	87
	R058206	1-4	1,600	80
	R058207	1-3	1,600	88
	R058209	1-2	1,600	94
R5	R058504	1-3	1,600	75
	R058508	1-4	1,600	70
	R058509	1-2	1,600	93
R6	R058602	1-3	1,600	74
	R058606	1-3	1,600	80
	R058608	1-4	1,600	79
R7	R058805	1-3	1,600	81
	R058807	1-4	1,600	70
	R058810	1-3	1,600	79
R8	R059106	1-4	1,600	72
	R059108	1-3	1,700	84
	R059109	1-3	1,600	81
	R059110	1-2	1,600	92
R10	R059504	1-3	1,600	74
	R059507	1-4	1,600	70
	R059509	1-3	1,600	80
	R059510	1-3	1,600	87

Score range and Cohen's kappa or intraclass correlation for the constructed-response items from the previous year assessed that were rescored in 2009, grade 4 reading combined national and state assessments, by item and block: 2011

Block	Item	Range of response codes	Sample size	Cohen’s Kappa	Intraclass correlation
† The intraclass correlation is not reported for dichotomously scored items. NOTE: Cohen's Kappa is a measure of reliability that is appropriate for items that are dichotomously scored. The Intraclass Correlation Coefficient is most appropriate for items with more than two categories. Special codes assigned to student responses including blank, off-task, and not-scorable were not included in the calculation of Cohen's Kappa or intraclass correlation. SOURCE: U.S. Department of Education, Institute of Education Sciences, National Center for Education Statistics, National Assessment of Educational Progress (NAEP), 2011 Reading Assessment.
R1	R057704	1-2	1,500	0.74	†
	R057705	1-3	1,500	0.74	0.82
	R057706	1-3	1,500	0.72	0.76
	R057707	1-4	1,500	0.66	0.82
	R057710	1-3	1,400	0.63	0.74
R2	R057806	1-4	1,400	0.68	0.81
	R057807	1-2	1,500	0.81	†
	R057809	1-3	1,400	0.68	0.74
R4	R058204	1-3	1,400	0.79	0.84
	R058206	1-4	1,400	0.66	0.83
	R058207	1-3	1,500	0.79	0.88
	R058209	1-2	1,400	0.82	†
R5	R058504	1-3	1,500	0.54	0.63
	R058508	1-4	1,400	0.55	0.80
	R058509	1-2	1,400	0.83	†
R6	R058602	1-3	1,400	0.56	0.65
	R058606	1-3	1,500	0.66	0.71
	R058608	1-4	1,400	0.65	0.84
R7	R058805	1-3	1,400	0.67	0.78
	R058807	1-4	1,500	0.55	0.78
	R058810	1-3	1,400	0.64	0.72
R8	R059106	1-4	1,500	0.52	0.76
	R059108	1-3	1,600	0.68	0.83
	R059109	1-3	1,500	0.66	0.74
	R059110	1-2	1,500	0.83	†
R10	R059504	1-3	1,500	0.52	0.66
	R059507	1-4	1,500	0.51	0.72
	R059509	1-3	1,400	0.66	0.77
	R059510	1-3	1,500	0.76	0.84

Printer-friendly Version