Table of Contents | Search Technical Documentation | References

NAEP Analysis and Scaling → Initial Activities → Constructed-Response Interrater Reliability → Score Range and Percentage of Exact Agreement for the Constructed-Response Items from Previous Year 2009, Rescored in 2015, Grade 12 Science National Assessment, by Item and Block: 2015

NAEP Technical DocumentationScore Range and Percentage of Exact Agreement for the Constructed-Response Items from Previous Year 2009, Rescored in 2015, Grade 12 Science National Assessment, by Item and Block: 2015

Score range and percentage of exact agreement for the constructed-response items from previous year 2009, rescored in 2015, grade 12 science national assessment, by item and block: 2015
Block	Item	Score range	Sample size	Percentage of exact agreement
S3	K133301	1-5	800	89
	K157201	1-3	700	86
	K157202	1-3	700	90
	K139001	1-3	800	83
	K126303	1-5	800	90
	K126304	1-3	800	88
	K126202	1-3	800	95
S4	K133801	1-5	800	76
	K128403	1-4	800	82
	K137201	1-4	800	92
	K157901	1-3	800	91
	K144401	1-4	800	88
S5	K129601	1-3	800	87
	K143903	1-3	800	88
	K131001	1-4	800	87
	K130301	1-3	800	83
	K158001	1-4	800	97
S6	K135001	1-4	800	90
	K158201	1-3	800	80
	K158202	1-3	800	93
	K158203	1-3	800	89
	K158204	1-3	800	93
	K139801	1-3	800	79
	K143801	1-3	800	96
	K134503	1-3	800	71
S7	K136101	1-4	800	85
	K136102	1-3	800	83
	K136103	1-4	800	90
	K132701	1-3	800	92
	K135401	1-3	800	93
	K158301	1-3	700	88
S8	K127201	1-5	800	65
	K137102	1-4	800	93
	K136401	1-3	800	90
	K142401	1-3	700	93
	K158401	1-3	1,200	87
	K158402	1-3	1,200	91
S11	K127701	1-4	800	85
	K125802	1-3	800	84
	K125803	1-3	800	97
	K144202	1-3	800	79
	K141701	1-4	800	90
S12	K129001	1-3	800	91
	K128701	1-5	800	64
	K133401	1-3	800	81
	K130001	1-4	800	85
	K130002	1-4	800	88
S13	K126001	1-5	800	74
	K140301	1-3	800	85
	K158701	1-3	800	96
	K158702	1-3	800	92
	K140303	1-3	800	87
	K138501	1-4	800	97
NOTE: Special codes assigned to student responses including blank, off-task, and not-scorable were included in the calculation of the percentage of exact agreement measure. SOURCE: U.S. Department of Education, Institute of Education Sciences, National Center for Education Statistics, National Assessment of Educational Progress (NAEP), 2015 Science Assessment.

Score range, Cohen's Kappa, and intraclass correlation for the constructed-response items from previous year 2009, rescored in 2015, grade 12 science national assessment, by item and block: 2015
Block	Item	Score range	Sample size	Cohen’s Kappa	Intraclass correlation
S3	K133301	1-5	800	.74	.92
	K157201	1-3	600	.76	.87
	K157202	1-3	600	.79	.84
	K139001	1-3	700	.65	.74
	K126303	1-5	700	.84	.96
	K126304	1-3	700	.81	.86
	K126202	1-3	700	.85	.93
S4	K133801	1-5	700	.60	.86
	K128403	1-4	700	.63	.73
	K137201	1-4	700	.84	.94
	K157901	1-3	700	.85	.90
	K144401	1-4	700	.77	.85
S5	K129601	1-3	700	.72	.83
	K143903	1-3	700	.72	.80
	K131001	1-4	800	.73	.76
	K130301	1-3	700	.66	.73
	K158001	1-4	700	.96	.96
S6	K135001	1-4	700	.69	.84
	K158201	1-3	700	.67	.82
	K158202	1-3	700	.82	.92
	K158203	1-3	700	.68	.81
	K158204	1-3	700	.75	.86
	K139801	1-3	700	.65	.79
	K143801	1-3	700	.83	.86
	K134503	1-3	700	.47	.65
S7	K136101	1-4	700	.74	.87
	K136102	1-3	700	.70	.78
	K136103	1-4	700	.82	.87
	K132701	1-3	700	.72	.81
	K135401	1-3	700	.88	.93
	K158301	1-3	700	.76	.84
S8	K127201	1-5	700	.49	.76
	K137102	1-4	700	.90	.97
	K136401	1-3	700	.75	.87
	K142401	1-3	700	.88	.94
	K158401	1-3	1,100	.77	.81
	K158402	1-3	1,000	.78	.85
S11	K127701	1-4	700	.76	.89
	K125802	1-3	700	.60	.71
	K125803	1-3	700	.93	.96
	K144202	1-3	700	.51	.64
	K141701	1-4	800	.85	.93
S12	K129001	1-3	700	.59	.60
	K128701	1-5	800	.46	.71
	K133401	1-3	800	.64	.72
	K130001	1-4	700	.81	.83
	K130002	1-4	700	.86	.90
S13	K126001	1-5	700	.54	.83
	K140301	1-3	700	.72	.82
	K158701	1-3	800	.92	.93
	K158702	1-3	800	.87	.91
	K140303	1-3	700	.79	.89
	K138501	1-4	700	.87	.98
NOTE: The intraclass correlation is most appropriate for items with more than two categories. Cohen's Kappa is a measure of reliability that is appropriate for both dichotomously and polytomously scored items. Special codes assigned to student responses including blank, off-task, and not-scorable were not included in the calculation of Cohen's Kappa or intraclass correlation. SOURCE: U.S. Department of Education, Institute of Education Sciences, National Center for Education Statistics, National Assessment of Educational Progress (NAEP), 2015 Science Assessment.

Last updated 15 November 2021 (ML)

Printer-friendly Version