NAEP Analysis and Scaling → Initial Activities → Constructed-Response Interrater Reliability → Range of Response Codes, Percent Agreement, and Cohen's Kappa or Intraclass Correlation for the Constructed-Response Items Used in Scaling, Age 17 Reading Long-Term Trend Assessment, by Item and Block: 2012

NAEP Technical DocumentationScore range and percent agreement for the constructed-response items used in scaling, age 17 reading long-term trend assessment, by item and block: 2012

Block	Item	Range of response codes	Sample size	Percent exact agreement
NOTE: Special codes assigned to student responses including blank, off-task, and not-scorable were included in the calculation of the percent exact agreement measure. SOURCE: U.S. Department of Education, Institute of Education Sciences, National Center for Education Statistics, National Assessment of Educational Progress (NAEP), 2012 Reading Long-Term Trend Assessment.
R1	N021203	1-5	600	80
R2	N023802	1-4	600	89
R3	N021803	1-4	600	89
R4	N022103	1-4	600	80
R5	N022405	1-4	600	85
R6	N025704	1-4	600	91
R8	N034203	1-5	600	76
R9	N029002	1-4	600	86

Score range and Cohen's Kappa or intraclass correlation for the constructed-response items used in scaling, age 17 reading long-term trend assessment, by item and block: 2012

Block	Item	Range of response codes	Sample size	Cohen’s Kappa	Intraclass correlation
† Not applicable. Cohen’s Kappa is a measure of reliability that is used for items that are dichotomously scored. Cohen’s Kappa is not reported for polytomously scored items. The intraclass correlation is used for items with more than two categories. The discrepancy in sample size for R9 compared to the first table is due to the fact that percent agreement is calculated from the entire sample size, while Cohen's Kappa and intraclass correlation statistics exclude those who omit the item. SOURCE: U.S. Department of Education, Institute of Education Sciences, National Center for Education Statistics, National Assessment of Educational Progress (NAEP), 2012 Reading Long-Term Trend Assessment.
R1	N021203	1-5	600	†	0.90
R2	N023802	1-4	600	†	0.87
R3	N021803	1-4	600	†	0.93
R4	N022103	1-4	600	†	0.77
R5	N022405	1-4	600	†	0.78
R6	N025704	1-4	600	†	0.94
R8	N034203	1-5	600	†	0.87
R9	N029002	1-4	500	†	0.87

Printer-friendly Version