NAEP Analysis and Scaling → Initial Activities → Constructed-Response Interrater Reliability → Range of Response Codes, Percent Agreement, and Cohen's Kappa or Intraclass Correlation for the Constructed-Response Items Used in Scaling, Age 17 Mathematics Long-Term Trend Assessment, by Item and Block: 2012

NAEP Technical DocumentationScore range and percent agreement for the constructed-response items used in scaling, age 17 mathematics long-term trend assessment, by item and block: 2012

Block	Item	Range of response codes	Sample size	Percent exact agreement
NOTE: Special codes assigned to student responses including blank, off-task, and not-scorable were included in the calculation of the percent exact agreement measure. SOURCE: U.S. Department of Education, Institute of Education Sciences, National Center for Education Statistics, National Assessment of Educational Progress (NAEP), 2012 Mathematics Long-Term Trend Assessment.
M1	N337501	1-2	1,000	100
	N332401	1-2	1,000	100
	N336701	1-2	1,000	100
	N337801	1-2	1,000	99
	N339101	1-2	1,000	100
M2	N308901	1-2	1,000	99
	N309301	1-2	1,000	100
	N309901	1-2	1,000	99
	N302901	1-2	1,000	100
	N310201	1-2	1,000	100
M3	N326601	1-2	1,000	99
	N326801	1-2	1,000	99
	N327101	1-2	1,000	100
	N325301	1-2	1,000	100
	N325601	1-2	1,000	99
	N325701	1-2	1,000	99
M4	N321001	1-2	1,000	100
	N321101	1-2	1,000	100
	N315501	1-2	1,000	99
	N321401	1-2	1,000	100
	N321901	1-2	1,000	100
M5	N303801	1-2	1,000	99
	N304101	1-2	1,000	100
	N304201	1-2	1,000	100
	N304601	1-2	1,000	99
	N304901	1-2	1,000	99
M6	N305501	1-2	1,000	99
	N305801	1-2	1,000	100
	N306201	1-2	1,000	99
	N306401	1-2	1,000	100
	N299001	1-2	1,000	100

Score range and Cohen's Kappa or intraclass correlation for the constructed-response items used in scaling, age 17 mathematics long-term trend assessment, by item and block: 2012

Block	Item	Range of response codes	Sample size	Cohen’s Kappa	Intraclass correlation
† Not applicable. The intraclass correlation is not reported for dichotomously scored items; Cohen's Kappa is not reported for polytomously scored items. NOTE: Cohen's Kappa is a measure of reliability that is appropriate for items that are dichotomously scored. The intraclass correlation coefficient is most appropriate for items with more than two categories. The discrepancy in sample sizes compared to the first table is due to the fact that percent agreement is calculated from the entire sample size, while Cohen's Kappa and intraclass correlation statistics exclude those who omit the item. SOURCE: U.S. Department of Education, Institute of Education Sciences, National Center for Education Statistics, National Assessment of Educational Progress (NAEP), 2012 Mathematics Long-Term Trend Assessment.
M1	N337501	1-2	935	1.00	†
	N332401	1-2	973	0.97	†
	N336701	1-2	862	1.00	†
	N337801	1-2	912	0.99	†
	N339101	1-2	820	1.00	†
M2	N308901	1-2	973	0.99	†
	N309301	1-2	975	1.00	†
	N309901	1-2	914	0.99	†
	N302901	1-2	965	0.99	†
	N310201	1-2	926	1.00	†
M3	N326601	1-2	974	0.93	†
	N326801	1-2	937	0.99	†
	N327101	1-2	926	1.00	†
	N325301	1-2	956	0.97	†
	N325601	1-2	927	0.98	†
	N325701	1-2	852	1.00	†
M4	N321001	1-2	966	0.99	†
	N321101	1-2	879	0.99	†
	N315501	1-2	876	0.99	†
	N321401	1-2	752	1.00	†
	N321901	1-2	690	1.00	†
M5	N303801	1-2	917	0.98	†
	N304101	1-2	967	0.99	†
	N304201	1-2	962	1.00	†
	N304601	1-2	912	0.99	†
	N304901	1-2	894	1.00	†
M6	N305501	1-2	917	0.99	†
	N305801	1-2	952	1.00	†
	N306201	1-2	864	1.00	†
	N306401	1-2	941	1.00	†
	N299001	1-2	946	1.00	†

Printer-friendly Version