Table of Contents | Search Technical Documentation | References

NAEP Analysis and Scaling → Initial Activities → Constructed-Response Interrater Reliability → Score Range and Percentage of Exact Agreement for the Constructed-Response Items from Previous Year 2011, Rescored in 2015, Grade 8 Science Combined National and State Assessment, by Item and Block: 2015

NAEP Technical DocumentationScore Range and Percentage of Exact Agreement for the Constructed-Response Items from Previous Year 2011, Rescored in 2015, Grade 8 Science Combined National and State Assessment, by Item and Block: 2015

Score range and percentage of exact agreement for the constructed-response items from previous year 2011, rescored in 2015, grade 8 science combined national and state assessment, by item and block: 2015
Block	Item	Score range	Sample size	Percentage of exact agreement
S3	K113901	1-3	1,700	87
	K113401	1-5	1,800	81
	K113201	1-4	1,600	90
	K113701	1-5	1,600	80
	K134201	1-4	1,600	92
S4	K114601	1-5	1,600	86
	K122402	1-4	1,900	84
	K122801	1-3	1,600	91
	K122602	1-3	1,600	85
	K125201	1-5	1,600	80
S5	K121301	1-4	1,900	89
	K120802	1-3	1,600	87
	K118501	1-4	1,700	88
	K154601	1-3	1,600	87
	K154701	1-5	1,600	83
S6	K123801	1-4	1,600	88
	K123802	1-3	1,600	86
	K122101	1-5	1,500	81
	K125003	1-3	1,600	83
	K125401	1-3	1,600	79
	K125402	1-3	1,600	84
S7	K110701	1-3	1,600	92
	K110702	1-3	1,600	91
	K110703	1-3	1,600	92
	K123901	1-5	1,600	77
	K110401	1-4	1,600	94
	K110501	1-3	1,600	93
	K121001	1-5	1,600	82
	K121002	1-3	1,900	82
	K121003	1-3	1,900	84
S8	K112401	1-4	1,600	93
	K155601	1-5	1,600	84
	K117202	1-2	1,900	94
S9	K115201	1-4	1,700	85
	K156101	1-4	1,600	89
	K114802	1-3	1,900	91
	K114801	1-4	1,700	85
	K124102	1-4	1,600	95
S12	K116901	1-3	1,900	83
	K123301	1-3	1,700	90
	K157001	1-3	1,800	72
	K119505	1-3	1,900	88
	K123401	1-4	1,700	83
	K112601	1-5	1,600	84
NOTE: Special codes assigned to student responses including blank, off-task, and not-scorable were included in the calculation of the percentage of exact agreement measure. SOURCE: U.S. Department of Education, Institute of Education Sciences, National Center for Education Statistics, National Assessment of Educational Progress (NAEP), 2015 Science Assessment.

Score range, Cohen's Kappa, and intraclass correlation for the constructed-response items from previous year 2011, rescored in 2015, grade 8 science combined national and state assessment, by item and block: 2015
Block	Item	Score range	Sample size	Cohen’s Kappa	Intraclass correlation
S3	K113901	1-3	1,500	.78	.86
	K113401	1-5	1,600	.72	.91
	K113201	1-4	1,400	.85	.94
	K113701	1-5	1,400	.71	.92
	K134201	1-4	1,400	.89	.96
S4	K114601	1-5	1,400	.80	.95
	K122402	1-4	1,700	.72	.90
	K122801	1-3	1,400	.83	.89
	K122602	1-3	1,400	.70	.84
	K125201	1-5	1,400	.72	.90
S5	K121301	1-4	1,700	.79	.94
	K120802	1-3	1,400	.75	.78
	K118501	1-4	1,500	.81	.91
	K154601	1-3	1,400	.78	.86
	K154701	1-5	1,400	.71	.90
S6	K123801	1-4	1,500	.82	.92
	K123802	1-3	1,400	.77	.86
	K122101	1-5	1,400	.73	.91
	K125003	1-3	1,400	.71	.80
	K125401	1-3	1,400	.60	.72
	K125402	1-3	1,400	.74	.83
S7	K110701	1-3	1,400	.85	.91
	K110702	1-3	1,400	.83	.90
	K110703	1-3	1,500	.80	.87
	K123901	1-5	1,500	.66	.88
	K110401	1-4	1,400	.91	.97
	K110501	1-3	1,400	.86	.91
	K121001	1-5	1,400	.70	.90
	K121002	1-3	1,700	.68	.78
	K121003	1-3	1,500	.67	.80
S8	K112401	1-4	1,400	.86	.92
	K155601	1-5	1,400	.72	.87
	K117202	1-2	1,700	.87	†
S9	K115201	1-4	1,600	.77	.91
	K156101	1-4	1,400	.77	.86
	K114802	1-3	1,700	.84	.92
	K114801	1-4	1,500	.71	.89
	K124102	1-4	1,400	.93	.97
S12	K116901	1-3	1,700	.70	.80
	K123301	1-3	1,500	.82	.87
	K157001	1-3	1,600	.48	.57
	K119505	1-3	1,700	.78	.85
	K123401	1-4	1,500	.73	.92
	K112601	1-5	1,400	.73	.82
† Not applicable. The intraclass correlation is not reported for dichotomously scored items. NOTE: The intraclass correlation is most appropriate for items with more than two categories. Cohen's Kappa is a measure of reliability that is appropriate for both dichotomously and polytomously scored items. Special codes assigned to student responses including blank, off-task, and not-scorable were not included in the calculation of Cohen's Kappa or intraclass correlation. SOURCE: U.S. Department of Education, Institute of Education Sciences, National Center for Education Statistics, National Assessment of Educational Progress (NAEP), 2015 Science Assessment.

Last updated 15 November 2021 (ML)

Printer-friendly Version