Table of Contents | Search Technical Documentation | References

NAEP Analysis and Scaling → Initial Activities → Constructed-Response Interrater Reliability → Score Range and Percentage of Exact Agreement for the Constructed-Response Items from Previous Year 2013, Rescored in 2015, Grade 12 Reading National Assessment, by Item and Block: 2015

NAEP Technical DocumentationScore Range and Percentage of Exact Agreement for the Constructed-Response Items from Previous Year 2013, Rescored in 2015, Grade 12 Reading National Assessment, by Item and Block: 2015

Score range and percentage of exact agreement for the constructed-response items from previous year 2013, rescored in 2015, grade 12 reading national assessment, by item and block: 2015
Block	Item	Score range	Sample size	Percentage of exact agreement
R1	R059604	1-2	1,100	92
	R059605	1-3	900	71
	R059606	1-3	1,100	88
	R059607	1-3	800	90
	R059610	1-4	800	81
R2	R069801	1-3	800	90
	R069804	1-3	1,100	83
	R069806	1-4	800	77
	R069810	1-3	800	81
R3	R060103	1-2	900	87
	R060105	1-2	1,100	94
	R060108	1-4	800	73
R5	R061602	1-3	1,100	88
	R061604	1-3	1,000	86
	R061606	1-3	800	77
	R061607	1-4	800	70
	R061608	1-2	800	90
R6	R078903	1-3	1,100	85
	R078905	1-4	800	76
	R078907	1-2	800	88
R7	R062203	1-3	800	85
	R062205	1-2	800	92
	R062207	1-4	800	74
	R062210	1-3	800	79
R8	R062303	1-3	800	80
	R062304	1-3	800	83
	R062308	1-4	800	74
	R062311	1-3	800	74
R9	R062501	1-3	800	81
	R062503	1-3	800	81
	R062505	1-3	800	78
	R062508	1-4	800	67
R10	R062705	1-3	800	79
	R062706	1-4	1,000	74
	R062708	1-2	1,100	93
	R062710	1-3	800	85
R12	R063101	1-3	800	79
	R063105	1-4	800	85
	R063109	1-3	1,000	94
	R063801	1-3	800	82
R13	R079101	1-3	1,100	80
	R079106	1-4	800	72
	R079107	1-2	800	89
	R079110	1-3	800	90
	R079111	1-3	800	81
NOTE: Special codes assigned to student responses including blank, off-task, and not-scorable were included in the calculation of the percentage of exact agreement measure. SOURCE: U.S. Department of Education, Institute of Education Sciences, National Center for Education Statistics, National Assessment of Educational Progress (NAEP), 2015 Reading Assessment.

Score range, Cohen's Kappa, and intraclass correlation for the constructed-response items from previous year 2013, rescored in 2015, grade 12 reading national assessment, by item and block: 2015
Block	Item	Score range	Sample size	Cohen’s Kappa	Intraclass correlation
R1	R059604	1-2	1,000	.62	†
	R059605	1-3	800	.49	.66
	R059606	1-3	900	.74	.83
	R059607	1-3	700	.76	.87
	R059610	1-4	700	.70	.87
R2	R069801	1-3	800	.81	.84
	R069804	1-3	1,000	.69	.79
	R069806	1-4	700	.63	.81
	R069810	1-3	700	.68	.77
R3	R060103	1-2	800	.61	†
	R060105	1-2	1,000	.80	†
	R060108	1-4	700	.58	.73
R5	R061602	1-3	1,000	.76	.89
	R061604	1-3	900	.73	.80
	R061606	1-3	800	.61	.80
	R061607	1-4	700	.52	.75
	R061608	1-2	700	.56	†
R6	R078903	1-3	1,000	.75	.86
	R078905	1-4	700	.61	.77
	R078907	1-2	700	.75	†
R7	R062203	1-3	700	.67	.68
	R062205	1-2	700	.85	†
	R062207	1-4	700	.57	.77
	R062210	1-3	700	.58	.67
R8	R062303	1-3	700	.62	.71
	R062304	1-3	700	.67	.76
	R062308	1-4	700	.59	.82
	R062311	1-3	700	.51	.73
R9	R062501	1-3	700	.60	.78
	R062503	1-3	700	.69	.81
	R062505	1-3	700	.60	.73
	R062508	1-4	700	.49	.72
R10	R062705	1-3	700	.62	.75
	R062706	1-4	900	.61	.83
	R062708	1-2	1,000	.78	†
	R062710	1-3	700	.74	.79
R12	R063101	1-3	700	.63	.79
	R063105	1-4	700	.74	.87
	R063109	1-3	900	.88	.96
	R063801	1-3	700	.69	.81
R13	R079101	1-3	1,000	.64	.80
	R079106	1-4	700	.57	.79
	R079107	1-2	800	.78	†
	R079110	1-3	700	.82	.90
	R079111	1-3	700	.68	.82
† Not applicable. The intraclass correlation is not reported for dichotomously scored items. NOTE: The intraclass correlation is most appropriate for items with more than two categories. Cohen's Kappa is a measure of reliability that is appropriate for both dichotomously and polytomously scored items. Special codes assigned to student responses including blank, off-task, and not-scorable were not included in the calculation of Cohen's Kappa or intraclass correlation. SOURCE: U.S. Department of Education, Institute of Education Sciences, National Center for Education Statistics, National Assessment of Educational Progress (NAEP), 2015 Reading Assessment.

Last updated 15 November 2021 (ML)

Printer-friendly Version