NAEP Analysis and Scaling → Initial Activities → Constructed-Response Interrater Reliability → Range of Response Codes, Percent Agreement, and Cohen's Kappa or Intraclass Correlation for the Constructed-Response Items Used in Scaling, Age 9 Mathematics Long-Term Trend Assessment, by Item and Block: 2012

NAEP Technical DocumentationScore range and percent agreement for the constructed-response items used in scaling, age 9 mathematics long-term trend assessment, by item and block: 2012

Block	Item	Range of response codes	Sample size	Percent exact agreement
NOTE: Special codes assigned to student responses including blank, off-task, and not-scorable were included in the calculation of the percent exact agreement measure. SOURCE: U.S. Department of Education, Institute of Education Sciences, National Center for Education Statistics, National Assessment of Educational Progress (NAEP), 2012 Mathematics Long-Term Trend Assessment.
M1	N331701	1-2	1,000	100
	N329501	1-2	1,000	100
	N329201	1-2	1,000	100
	N331501	1-2	1,000	100
	N329101	1-2	1,000	99
	N331601	1-2	1,000	100
	N331901	1-2	1,000	100
	N332101	1-2	1,000	100
M2	N294201	1-2	1,000	100
	N294301	1-2	1,000	100
	N295301	1-2	1,000	100
	N295401	1-2	1,000	100
	N295701	1-2	1,000	99
M3	N322401	1-2	1,100	100
	N323101	1-2	1,100	100
	N323701	1-2	1,100	100
	N323901	1-2	1,100	100
	N324001	1-2	1,100	99
M4	N312901	1-2	1,100	100
	N313001	1-2	1,100	100
	N313101	1-2	1,100	100
	N313102	1-2	1,100	99
	N313103	1-2	1,100	100
	N313701	1-2	1,100	100
	N313702	1-2	1,100	100
	N313801	1-2	1,100	99
	N313901	1-2	1,100	100
M5	N287601	1-2	1,000	99
	N288101	1-2	1,000	100
	N288501	1-2	1,000	100
	N289201	1-2	1,000	99
M6	N289701	1-2	1,000	100
	N289901	1-2	1,000	100
	N290101	1-2	1,000	100
	N290701	1-2	1,000	100
	N291501	1-2	1,000	100

Score range and Cohen's Kappa or intraclass correlation for the constructed-response items used in scaling, age 9 mathematics long-term trend assessment, by item and block: 2012

Block	Item	Range of response codes	Sample size	Cohen’s Kappa	Intraclass correlation
† Not applicable. The intraclass correlation is not reported for dichotomously scored items; Cohen’s Kappa is not reported for polytomously scored items. NOTE:  Cohen's Kappa is a measure of reliability that is used for items that are dichotomously scored. The intraclass correlation is used for items with more than two categories. The discrepancy in sample sizes compared to the first table is due to the fact that percent agreement is calculated from the entire sample size, while Cohen's Kappa and intraclass correlation statistics exclude those who omit the item. SOURCE: U.S. Department of Education, Institute of Education Sciences, National Center for Education Statistics, National Assessment of Educational Progress (NAEP), 2012 Mathematics Long-term Trend Assessment.
M1	N331701	1-2	1,000	0.99	†
	N329501	1-2	1,000	0.99	†
	N329201	1-2	1,000	0.99	†
	N331501	1-2	1,000	1.00	†
	N329101	1-2	1,000	0.99	†
	N331601	1-2	900	0.98	†
	N331901	1-2	900	1.00	†
	N332101	1-2	800	1.00	†
M2	N294201	1-2	1,000	0.98	†
	N294301	1-2	1,000	1.00	†
	N295301	1-2	1,000	0.99	†
	N295401	1-2	1,000	1.00	†
	N295701	1-2	900	0.99	†
M3	N322401	1-2	1,000	1.00	†
	N323101	1-2	1,000	0.99	†
	N323701	1-2	900	1.00	†
	N323901	1-2	900	1.00	†
	N324001	1-2	900	0.99	†
M4	N312901	1-2	1,100	0.99	†
	N313001	1-2	1,100	0.99	†
	N313101	1-2	1,100	1.00	†
	N313102	1-2	1,100	0.99	†
	N313103	1-2	1,100	0.99	†
	N313701	1-2	1,000	1.00	†
	N313702	1-2	900	1.00	†
	N313801	1-2	900	0.99	†
	N313901	1-2	900	1.00	†
M5	N287601	1-2	1,000	0.98	†
	N288101	1-2	1,000	1.00	†
	N288501	1-2	1,000	0.99	†
	N289201	1-2	900	0.99	†
M6	N289701	1-2	1,000	0.99	†
	N289901	1-2	1,000	1.00	†
	N290101	1-2	1,000	0.99	†
	N290701	1-2	1,000	1.00	†
	N291501	1-2	900	1.00	†

Printer-friendly Version