Table of Contents | Search Technical Documentation | References

NAEP Analysis and Scaling → Initial Activities → Constructed-Response Interrater Reliability → Score Range and Percentage of Exact Agreement for the Constructed-Response Items Used in Scaling, Grade 8 Mathematics Combined National and State Assessment, by Item and Block: 2015

NAEP Technical DocumentationScore Range and Percentage of Exact Agreement for the Constructed-Response Items Used in Scaling, Grade 8 Mathematics Combined National and State Assessment, by Item and Block: 2015

Score range and percentage of exact agreement for the constructed-response items used in scaling, grade 8 mathematics combined national and state assessment, by item and block: 2015
Block	Item	Score range	Sample size	Percentage of exact agreement
M3	M350801	1-3	1,300	89
	M233601	1-3	1,300	98
	M351101	1-2	1,300	100
	M351102	1-2	1,300	99
	M351103	1-2	1,300	99
	M349501	1-3	1,300	93
	M349801	1-3	1,300	94
	M349802	1-3	1,300	97
M4	M151501	1-3	1,300	94
	M151601	1-3	1,300	96
	M152602	1-5	1,300	97
M5	M221203	1-3	1,400	98
	M221204	1-3	1,400	85
	M221701	1-2	1,400	99
	M222301	1-3	1,400	99
M6	M351901	1-2	1,300	99
	M351902	1-2	1,300	99
	M352301	1-3	1,300	99
	M353401	1-3	1,300	99
	M232101	1-2	1,300	99
M7	M235701	1-2	1,600	99
	M355301	1-3	1,600	98
	M356401	1-2	1,600	98
	M356601	1-2	1,600	98
	M356602	1-2	1,600	97
	M356603	1-2	1,600	97
	M356901	1-2	1,600	100
M8	M239001	1-3	1,600	98
	M237901	1-3	1,600	99
	M317301	1-3	1,600	98
	M317302	1-3	1,600	96
M9	M235501	1-2	1,500	99
	M234702	1-3	1,500	99
	M237201	1-2	1,500	100
	M237202	1-3	1,500	99
	M237203	1-3	1,500	99
	M237204	1-3	1,500	99
M10	M222701	1-3	1,400	97
	M223101	1-3	1,400	98
	M223301	1-3	1,400	91
	M223801	1-3	1,400	98
M11	M163101	1-3	1,300	95
	M168301	1-3	1,300	96
	M162901	1-3	1,300	97
	M167901	1-3	1,300	97
	M104901	1-3	1,300	95
M12	M233501	1-2	1,300	99
	M234001	1-2	1,300	100
	M231501	1-2	1,300	99
	M234201	1-3	1,300	92
NOTE: Special codes assigned to student responses including blank, off-task, and not-scorable were included in the calculation of the percentage of exact agreement measure. SOURCE: U.S. Department of Education, Institute of Education Sciences, National Center for Education Statistics, National Assessment of Educational Progress (NAEP), 2015 Mathematics Assessment.

Score range, Cohen's Kappa, and intraclass correlation for the constructed-response items used in scaling, grade 8 mathematics combined national and state assessment, by item and block: 2015
Block	Item	Score range	Sample size	Cohen’s Kappa	Intraclass correlation
M3	M350801	1-3	1,300	.82	.89
	M233601	1-3	1,300	.96	.98
	M351101	1-2	1,300	.98	†
	M351102	1-2	1,300	.97	†
	M351103	1-2	1,300	.99	†
	M349501	1-3	1,200	.87	.88
	M349801	1-3	1,200	.89	.90
	M349802	1-3	1,100	.94	.95
M4	M151501	1-3	1,300	.91	.96
	M151601	1-3	1,200	.88	.95
	M152602	1-5	1,200	.96	.98
M5	M221203	1-3	1,300	.96	.97
	M221204	1-3	1,300	.71	.76
	M221701	1-2	1,300	.99	†
	M222301	1-3	1,100	.99	.99
M6	M351901	1-2	1,300	1.00	†
	M351902	1-2	1,300	.99	†
	M352301	1-3	1,300	.98	.99
	M353401	1-3	1,200	.99	.99
	M232101	1-2	1,100	.99	†
M7	M235701	1-2	1,500	.99	†
	M355301	1-3	1,500	.96	.98
	M356401	1-2	1,500	.99	†
	M356601	1-2	1,400	.98	†
	M356602	1-2	1,400	.98	†
	M356603	1-2	1,400	.97	†
	M356901	1-2	1,400	1.00	†
M8	M239001	1-3	1,600	.97	.98
	M237901	1-3	1,500	.99	.99
	M317301	1-3	1,400	.96	.98
	M317302	1-3	1,100	.83	.91
M9	M235501	1-2	1,400	.99	†
	M234702	1-3	1,400	.99	1.00
	M237201	1-2	1,300	.99	†
	M237202	1-3	1,200	.99	1.00
	M237203	1-3	1,200	.96	.97
	M237204	1-3	1,200	.98	.99
M10	M222701	1-3	1,300	.95	.97
	M223101	1-3	1,400	.98	.99
	M223301	1-3	1,300	.86	.90
	M223801	1-3	1,200	.97	.98
M11	M163101	1-3	1,300	.91	.95
	M168301	1-3	1,200	.94	.98
	M162901	1-3	1,300	.96	.97
	M167901	1-3	1,200	.91	.96
	M104901	1-3	1,100	.89	.92
M12	M233501	1-2	1,300	.98	†
	M234001	1-2	1,300	.99	†
	M231501	1-2	1,300	.98	†
	M234201	1-3	1,100	.82	.90
† Not applicable. The intraclass correlation is not reported for dichotomously scored items. NOTE: The intraclass correlation is most appropriate for items with more than two categories. Cohen's Kappa is a measure of reliability that is appropriate for both dichotomously and polytomously scored items. Special codes assigned to student responses including blank, off-task, and not-scorable were not included in the calculation of Cohen's Kappa or intraclass correlation. SOURCE: U.S. Department of Education, Institute of Education Sciences, National Center for Education Statistics, National Assessment of Educational Progress (NAEP), 2015 Mathematics Assessment.

Last updated 09 November 2021 (ML)

Printer-friendly Version