Table of Contents | Search Technical Documentation | References

NAEP Analysis and Scaling → Initial Activities → Constructed-Response Interrater Reliability → Score Range and Percentage of Exact Agreement for the Constructed-Response Items Used in Scaling, Grade 8 Science Combined National and State Assessment, by Item and Block: 2015

NAEP Technical DocumentationScore Range and Percentage of Exact Agreement for the Constructed-Response Items Used in Scaling, Grade 8 Science Combined National and State Assessment, by Item and Block: 2015

Score range and percentage of exact agreement for the constructed-response items used in scaling, grade 8 science combined national and state assessment, by item and block: 2015
Block	Item	Score range	Sample size	Percentage of exact agreement
S3	K113901	1-3	1,100	85
	K113401	1-5	1,100	87
	K113201	1-4	1,100	89
	K113701	1-5	1,100	81
	K134201	1-4	1,100	93
S4	K114601	1-5	1,100	90
	K122402	1-4	1,100	85
	K122801	1-3	1,100	93
	K122602	1-3	1,100	87
	K125201	1-5	1,100	83
S5	K121301	1-4	1,100	92
	K120802	1-3	1,100	89
	K118501	1-4	1,100	86
	K154601	1-3	1,100	92
	K154701	1-5	1,100	88
S6	K123801	1-4	1,100	86
	K123802	1-3	1,100	86
	K122101	1-5	1,100	84
	K125003	1-3	1,100	86
	K125401	1-3	1,100	85
	K125402	1-3	1,100	84
S7	K110701	1-3	1,100	90
	K110702	1-3	1,100	92
	K110703	1-3	1,100	91
	K123901	1-5	1,100	77
	K110401	1-4	1,100	94
	K110501	1-3	1,100	93
	K121001	1-5	1,100	89
	K121002	1-3	1,100	87
	K121003	1-3	1,100	90
S8	K112401	1-4	1,100	96
	K155601	1-5	1,100	88
	K117202	1-2	1,100	95
	K155751	1-3	1,100	88
	K155752	1-4	1,100	88
S9	K115201	1-4	1,100	84
	K156101	1-4	1,100	93
	K114802	1-3	1,100	88
	K114801	1-4	1,100	87
	K124102	1-4	1,100	97
S10	K172101	1-3	1,100	90
	K167601	1-4	1,100	90
	K167701	1-3	1,100	89
S11	K169701	1-4	1,100	92
	K168701	1-3	1,100	90
	K166101	1-3	1,100	90
	K171301	1-3	1,100	87
S12	K116901	1-3	1,100	83
	K123301	1-3	1,100	93
	K157001	1-3	1,100	74
	K119505	1-3	1,100	85
	K123401	1-4	1,100	84
	K112601	1-5	1,100	90
NOTE: Special codes assigned to student responses including blank, off-task, and not-scorable were included in the calculation of the percentage of exact agreement measure. SOURCE: U.S. Department of Education, Institute of Education Sciences, National Center for Education Statistics, National Assessment of Educational Progress (NAEP), 2015 Science Assessment.

Score range, Cohen's Kappa, and intraclass correlation for the constructed-response items used in scaling, grade 8 science combined national and state assessment, by item and block: 2015
Block	Item	Score range	Sample size	Cohen’s Kappa	Intraclass correlation
S3	K113901	1-3	1,100	.77	.86
	K113401	1-5	1,000	.80	.94
	K113201	1-4	1,100	.85	.94
	K113701	1-5	1,000	.72	.90
	K134201	1-4	900	.91	.97
S4	K114601	1-5	1,000	.85	.97
	K122402	1-4	1,100	.74	.89
	K122801	1-3	900	.84	.89
	K122602	1-3	1,000	.73	.85
	K125201	1-5	900	.76	.92
S5	K121301	1-4	1,100	.77	.88
	K120802	1-3	1,100	.81	.84
	K118501	1-4	1,100	.81	.91
	K154601	1-3	1,000	.87	.91
	K154701	1-5	1,000	.75	.87
S6	K123801	1-4	1,100	.82	.91
	K123802	1-3	1,100	.77	.85
	K122101	1-5	1,000	.76	.91
	K125003	1-3	1,000	.76	.85
	K125401	1-3	900	.70	.76
	K125402	1-3	900	.72	.83
S7	K110701	1-3	1,100	.83	.90
	K110702	1-3	1,100	.87	.91
	K110703	1-3	1,100	.81	.89
	K123901	1-5	1,100	.67	.88
	K110401	1-4	900	.91	.97
	K110501	1-3	1,000	.85	.89
	K121001	1-5	800	.73	.89
	K121002	1-3	700	.70	.81
	K121003	1-3	600	.73	.85
S8	K112401	1-4	1,100	.91	.95
	K155601	1-5	1,100	.73	.90
	K117202	1-2	1,100	.89	†
	K155751	1-3	1,100	.73	.79
	K155752	1-4	1,000	.78	.89
S9	K115201	1-4	1,100	.76	.90
	K156101	1-4	1,100	.78	.86
	K114802	1-3	1,100	.81	.91
	K114801	1-4	1,000	.75	.90
	K124102	1-4	1,100	.96	.98
S10	K172101	1-3	1,100	.84	.89
	K167601	1-4	1,100	.73	.79
	K167701	1-3	1,000	.81	.87
S11	K169701	1-4	1,100	.83	.91
	K168701	1-3	1,100	.83	.83
	K166101	1-3	1,100	.79	.82
	K171301	1-3	1,100	.77	.89
S12	K116901	1-3	1,100	.70	.79
	K123301	1-3	1,100	.86	.90
	K157001	1-3	1,100	.54	.62
	K119505	1-3	1,100	.73	.80
	K123401	1-4	1,100	.77	.93
	K112601	1-5	900	.73	.76
† Not applicable. The intraclass correlation is not reported for dichotomously scored items. NOTE: The intraclass correlation is most appropriate for items with more than two categories. Cohen's Kappa is a measure of reliability that is appropriate for both dichotomously and polytomously scored items. Special codes assigned to student responses including blank, off-task, and not-scorable were not included in the calculation of Cohen's Kappa or intraclass correlation. SOURCE: U.S. Department of Education, Institute of Education Sciences, National Center for Education Statistics, National Assessment of Educational Progress (NAEP), 2015 Science Assessment.

Last updated 15 November 2021 (ML)

Printer-friendly Version