NAEP Analysis and Scaling → Initial Activities → Constructed-Response Interrater Reliability → Range of Response Codes, Percent Agreement, and Cohen's Kappa or Intraclass Correlation for the Constructed-Response Items Used in Scaling, Age 13 Reading Long-Term Trend Assessment, by Item and Block: 2012

NAEP Technical DocumentationScore range and percent agreement for the constructed-response items used in scaling, age 13 reading long-term trend assessment, by item and block: 2012

Block	Item	Range of response codes	Sample size	Percent exact agreement
NOTE: Special codes assigned to student responses including blank, off-task, and not-scorable were included in the calculation of the percent exact agreement measure. SOURCE: U.S. Department of Education, Institute of Education Sciences, National Center for Education Statistics, National Assessment of Educational Progress (NAEP), 2012 Reading Long-Term Trend Assessment.
R1	N021203	1-5	600	79
R2	N023802	1-4	600	86
R3	N021803	1-4	600	92
R4	N022103	1-4	600	87
R5	N022405	1-4	600	88
R8	N034203	1-5	600	79
R9	N029002	1-4	600	86

Score range and Cohen's Kappa or intraclass correlation for the constructed-response items used in scaling, age 13 reading long-term trend assessment, by item and block: 2012

Block	Item	Range of response codes	Sample size	Cohen’s Kappa	Intraclass correlation
NOTE: Cohen’s Kappa is a measure of reliability that is used for items that are dichotomously scored. The intraclass correlation is used for items with more than two categories. SOURCE: U.S. Department of Education, Institute of Education Sciences, National Center for Education Statistics, National Assessment of Educational Progress (NAEP), 2012 Reading Long-Term Trend Assessment.
R1	N021203	1-5	600	0.70	0.89
R2	N023802	1-4	600	0.76	0.86
R3	N021803	1-4	600	0.88	0.93
R4	N022103	1-4	600	0.78	0.85
R5	N022405	1-4	600	0.78	0.84
R8	N034203	1-5	600	0.71	0.89
R9	N029002	1-4	600	0.75	0.86

Printer-friendly Version