NAEP Analysis and Scaling → Initial Activities → Constructed-Response Interrater Reliability → Range of Response Codes, Percent Agreement, and Cohen's Kappa or Intraclass Correlation for the Constructed-Response Items Used in Scaling, Age 9 Reading Long-Term Trend Assessment, by Item and Block: 2012

NAEP Technical DocumentationScore range and percent agreement for the constructed-response items used in scaling, age 9 reading long-term trend assessment, by item and block: 2012

Block	Item	Range of response codes	Sample size	Percent exact agreement
NOTE: Special codes assigned to student responses including blank, off-task, and not-scorable were included in the calculation of the percent exact agreement measure. SOURCE: U.S. Department of Education, Institute of Education Sciences, National Center for Education Statistics, National Assessment of Educational Progress (NAEP), 2012 Reading Long-Term Trend Assessment.
R1	N021203	1-5	600	83
R3	N021803	1-4	600	95
R4	N022103	1-4	600	87
R5	N022405	1-4	600	88

Score range and Cohen's Kappa or intraclass correlation for the constructed-response items used in scaling, age 9 reading long-term trend assessment, by item and block: 2012

Block	Item	Range of response codes	Sample size	Cohen’s Kappa	Intraclass correlation
NOTE: Cohen’s Kappa is a measure of reliability that is used for items that are dichotomously scored. The intraclass correlation is used for items with more than two categories. SOURCE: U.S. Department of Education, Institute of Education Sciences, National Center for Education Statistics, National Assessment of Educational Progress (NAEP), 2012 Reading Long-Term Trend Assessment.
R1	N021203	1-5	600	0.74	0.90
R3	N021803	1-4	600	0.89	0.92
R4	N022103	1-4	600	0.77	0.84
R5	N022405	1-4	600	0.78	0.86

Printer-friendly Version