NAEP Analysis and Scaling → Initial Activities → Constructed-Response Interrater Reliability → Range of Response Codes, Percent Agreement, and Cohen's Kappa or Intraclass Correlation for the Constructed-Response Items From Previous Years, Rescored in 2012, Age 13 Reading Long-Term Trend Assessment, by Item and Block: 2012

NAEP Technical DocumentationScore range, percent agreement for the constructed-response items from the previous year assessed that were rescored in 2012, age 13 reading long-term trend assessment, by block and item: 2012

Block	Item	Range of response codes	Sample size	Percent exact agreement
NOTE: Special codes assigned to student responses including blank, off-task, and not-scorable were included in the calculation of the percent exact agreement measure. SOURCE: U.S. Department of Education, Institute of Education Sciences, National Center for Education Statistics, National Assessment of Educational Progress (NAEP), 2012 Reading Long-Term Trend Assessment.
RA	N021203	1-5	600	69
RC	N021803	1-4	600	87
RD	N022103	1-4	600	79
RE	N022405	1-4	600	84
RB	N023802	1-4	600	83
RI	N029002	1-4	600	80

Score range and Cohen's Kappa or intraclass correlation for the constructed-response items from the previous year assessed that were rescored in 2012, age 13 reading long-term trend assessment, by block and item: 2012

Block	Item	Range of response codes	Sample size	Cohen’s Kappa	Intraclass correlation
NOTE: Cohen’s Kappa is a measure of reliability that is used for items that are dichotomously scored. The intraclass correlation is used for items with more than two categories. The intraclass correlation is not reported for dichotomously scored items. SOURCE: U.S. Department of Education, Institute of Education Sciences, National Center for Education Statistics, National Assessment of Educational Progress (NAEP), 2012 Reading Long-Term Trend Assessment.
RA	N021203	1-5	600	0.57	0.82
RC	N021803	1-4	600	0.79	0.87
RD	N022103	1-4	600	0.66	0.78
RE	N022405	1-4	600	0.71	0.80
RB	N023802	1-4	600	0.69	0.81
RI	N029002	1-4	600	0.64	0.79

Printer-friendly Version