NAEP Analysis and Scaling → Initial Activities → Constructed-Response Interrater Reliability → Range of Response Codes, Percent Agreement, and Cohen's Kappa or Intraclass Correlation for the Constructed-Response Items Used in Scaling, Grade 8 Science Combined National and State Assessments, by Item and Block: 2011

NAEP Technical DocumentationScore range and percent agreement for the constructed-response items used in scaling, grade 8 science combined national and state assessments, by item and block: 2011

Block	Item	Range of response codes	Sample size	Percent exact agreement
NOTE: Special codes assigned to student responses including blank, off-task, and not-scorable were included in the calculation of the Percent Exact Agreement. SOURCE: U.S. Department of Education, Institute of Education Sciences, National Center for Education Statistics, National Assessment of Educational Progress (NAEP), 2011 Science Assessment.
S3	K113201	1 - 4	1500	90
	K113401	1 - 5	1500	86
	K113701	1 - 5	1500	78
	K113901	1 - 3	1500	86
	K134201	1 - 4	1500	94
S4	K114601	1 - 5	1400	90
	K122402	1 - 4	1400	82
	K122602	1 - 3	1400	85
	K122801	1 - 3	1400	89
	K125201	1 - 5	1400	83
S5	K118501	1 - 4	1400	91
	K120802	1 - 3	1400	87
	K121301	1 - 4	1400	94
	K154601	1 - 3	1400	87
	K154701	1 - 5	1400	86
S6	K122101	1 - 5	1400	81
	K123801	1 - 4	1400	88
	K123802	1 - 3	1400	89
	K125003	1 - 3	1400	85
	K125401	1 - 3	1400	89
	K125402	1 - 3	1400	88
S7	K110401	1 - 4	1400	95
	K110501	1 - 3	1400	93
	K110701	1 - 3	1400	91
	K110702	1 - 3	1400	90
	K110703	1 - 3	1400	93
	K121001	1 - 5	1400	86
	K121002	1 - 3	1400	84
	K121003	1 - 3	1400	88
	K123901	1 - 5	1300	80
S8	K112401	1 - 4	1400	93
	K117202	1 - 2	1400	95
	K155601	1 - 5	1400	89
	K155701	1 - 3	1400	85
	K155702	1 - 4	1400	86
S9	K114801	1 - 4	1400	90
	K114802	1 - 3	1400	94
	K115201	1 - 4	1400	85
	K124102	1 - 4	1400	96
	K156101	1 - 4	1500	93
S11	K111801	1 - 3	1400	89
	K111802	1 - 3	1400	90
	K111803	1 - 3	1400	92
	K117001	1 - 4	1400	98
	K117003	1 - 2	1400	92
	K117004	1 - 3	1400	92
	K119902	1 - 4	1400	88
	K123601	1 - 5	1400	90
S12	K112601	1 - 5	1400	91
	K116901	1 - 3	1400	83
	K119505	1 - 3	1400	89
	K123301	1 - 3	1300	92
	K123401	1 - 4	1400	83
	K157001	1 - 3	1400	76

Score range and Cohen's Kappa or intraclass correlation for the constructed-response items used in scaling,  grade 8 science combined national and state assessments, by item and block: 2011

Block	Item	Range of response codes	Sample size	Cohen’s Kappa	Intraclass correlation
† The intraclass correlation is not reported for dichotomously scored items. NOTE: Cohen's Kappa is a measure of reliability that is appropriate for items that are dichotomously scored. The Intraclass Correlation Coefficient is most appropriate for items with more than two categories. Special codes assigned to student responses including blank, off-task, and not-scorable were not included in the calculation of Cohen's Kappa or intraclass correlation. SOURCE: U.S. Department of Education, Institute of Education Sciences, National Center for Education Statistics, National Assessment of Educational Progress (NAEP), 2011 Science Assessment.
S3	K113201	1 - 4	1400	0.86	0.94
	K113401	1 - 5	1300	0.79	0.93
	K113701	1 - 5	1300	0.70	0.90
	K113901	1 - 3	1400	0.80	0.86
	K134201	1 - 4	1300	0.92	0.97
S4	K114601	1 - 5	1200	0.87	0.97
	K122402	1 - 4	1400	0.73	0.88
	K122602	1 - 3	1300	0.73	0.83
	K122801	1 - 3	1100	0.80	0.86
	K125201	1 - 5	1200	0.75	0.93
S5	K118501	1 - 4	1400	0.88	0.94
	K120802	1 - 3	1300	0.76	0.79
	K121301	1 - 4	1300	0.85	0.93
	K154601	1 - 3	1300	0.80	0.87
	K154701	1 - 5	1200	0.73	0.86
S6	K122101	1 - 5	1300	0.73	0.90
	K123801	1 - 4	1300	0.84	0.94
	K123802	1 - 3	1300	0.83	0.89
	K125003	1 - 3	1200	0.77	0.85
	K125401	1 - 3	1200	0.77	0.80
	K125402	1 - 3	1200	0.81	0.90
S7	K110401	1 - 4	1200	0.92	0.98
	K110501	1 - 3	1200	0.87	0.89
	K110701	1 - 3	1300	0.85	0.91
	K110702	1 - 3	1300	0.84	0.90
	K110703	1 - 3	1300	0.85	0.90
	K121001	1 - 5	1100	0.67	0.83
	K121002	1 - 3	900	0.64	0.73
	K121003	1 - 3	800	0.70	0.80
	K123901	1 - 5	1300	0.70	0.87
S8	K112401	1 - 4	1300	0.85	0.90
	K117202	1 - 2	1300	0.88	†
	K155601	1 - 5	1300	0.76	0.88
	K155701	1 - 3	1300	0.66	0.73
	K155702	1 - 4	1200	0.71	0.83
S9	K114801	1 - 4	1300	0.80	0.91
	K114802	1 - 3	1300	0.91	0.95
	K115201	1 - 4	1300	0.77	0.92
	K124102	1 - 4	1300	0.94	0.97
	K156101	1 - 4	1400	0.78	0.87
S11	K111801	1 - 3	1200	0.79	0.86
	K111802	1 - 3	1300	0.81	0.90
	K111803	1 - 3	1200	0.80	0.88
	K117001	1 - 4	1300	0.96	0.98
	K117003	1 - 2	1300	0.77	†
	K117004	1 - 3	1100	0.81	0.91
	K119902	1 - 4	1100	0.78	0.82
	K123601	1 - 5	1300	0.87	0.96
S12	K112601	1 - 5	1100	0.74	0.80
	K116901	1 - 3	1300	0.73	0.83
	K119505	1 - 3	1300	0.81	0.87
	K123301	1 - 3	1300	0.86	0.89
	K123401	1 - 4	1300	0.76	0.92
	K157001	1 - 3	1300	0.56	0.63

Printer-friendly Version