Table of Contents | Search Technical Documentation | References

NAEP Analysis and Scaling → Initial Activities → Constructed-Response Interrater Reliability → Score Range and Percentage of Exact Agreement for the Constructed-Response Items Used in Scaling, Grade 8 Visual Arts National Assessment, by Block and Item: 2016

NAEP Technical DocumentationScore Range and Percentage of Exact Agreement for the Constructed-Response Items Used in Scaling, Grade 8 Visual Arts National Assessment, by Block and Item: 2016

Score range and percentage of exact agreement for the constructed-response items used in scaling, grade 8 visual arts national assessment, by block and item: 2016
Block	Item	Score range	Sample size	Percentage of exact agreement
V3	VC00002	1-3	600	89
	VC00005	1-4	500	78
	VC00007	1-4	500	84
	VC000A4	1-3	600	83
	VC000A6	1-2	500	97
	VC000B4	1-3	600	74
	VC000B6	1-4	500	82
V4	VD00001	1-2	500	93
	VD00007	1-3	500	73
	VD00008	1-4	500	74
	VD00009	1-4	500	73
	VD000A5	1-3	500	74
	VD000A6	1-3	500	77
	VD000B5	1-3	500	77
	VD000B6	1-2	500	89
V5	VE00001	1-2	600	89
	VE00002	1-4	600	77
	VE00004	1-3	600	79
	VE00008	1-3	600	79
	VE00009	1-5	600	83
	VE00010	1-3	600	87
	VE00303	1-3	600	77
V6	VF00001	1-3	600	78
	VF00004	1-3	600	81
	VF000A6	1-4	600	79
	VF000A7	1-3	600	88
	VF000B6	1-4	600	77
	VF000B7	1-2	600	84
	VF000B8	1-4	600	77
	VF000C6	1-4	600	71
	VF000C7	1-3	600	81
NOTE: Special codes assigned to student responses including omitted, off-task, and illegible were included in the calculation of the percentage of exact agreement measure. SOURCE: U.S. Department of Education, Institute of Education Sciences, National Center for Education Statistics, National Assessment of Educational Progress (NAEP), 2016 Visual Arts Assessment.

Score range, Cohen's Kappa, and intraclass correlation for the constructed-response items used in scaling, grade 8 visual arts national assessment, by block and item: 2016
Block	Item	Score range	Sample size	Cohen’s Kappa	Intraclass correlation
V3	VC00002	1-3	500	.83	.90
	VC00005	1-4	500	.53	.65
	VC00007	1-4	400	.66	.84
	VC000A4	1-3	500	.72	.83
	VC000A6	1-2	500	.92	†
	VC000B4	1-3	500	.55	.67
	VC000B6	1-4	500	.67	.79
V4	VD00001	1-2	500	.85	†
	VD00007	1-3	500	.50	.64
	VD00008	1-4	500	.49	.65
	VD00009	1-4	500	.51	.69
	VD000A5	1-3	500	.56	.66
	VD000A6	1-3	500	.60	.70
	VD000B5	1-3	500	.64	.79
	VD000B6	1-2	500	.77	†
V5	VE00001	1-2	600	.75	†
	VE00002	1-4	600	.51	.65
	VE00004	1-3	500	.62	.70
	VE00008	1-3	500	.64	.78
	VE00009	1-5	600	.74	.86
	VE00010	1-3	400	.67	.72
	VE00303	1-3	600	.57	.66
V6	VF00001	1-3	600	.63	.73
	VF00004	1-3	600	.67	.75
	VF000A6	1-4	500	.68	.81
	VF000A7	1-3	500	.80	.85
	VF000B6	1-4	600	.65	.76
	VF000B7	1-2	500	.62	†
	VF000B8	1-4	600	.65	.81
	VF000C6	1-4	600	.56	.72
	VF000C7	1-3	500	.71	.79
† Not applicable. The intraclass correlation is not reported for dichotomously scored items. NOTE: The intraclass correlation is most appropriate for items with more than two categories. Cohen's Kappa is a measure of reliability that is appropriate for both dichotomously and polytomously scored items. Special codes assigned to student responses including omitted, off-task, and illegible were not included in the calculation of Cohen's Kappa or intraclass correlation. SOURCE: U.S. Department of Education, Institute of Education Sciences, National Center for Education Statistics, National Assessment of Educational Progress (NAEP), 2016 Visual Arts Assessment.

Last updated 14 February 2022 (SK)

Printer-friendly Version