Table of Contents | Search Technical Documentation | References

NAEP Analysis and Scaling → Initial Activities → Constructed-Response Interrater Reliability → Score Range and Percentage of Exact Agreement for the Constructed-Response Items From Previous Year 2008, Rescored in 2016, Grade 8 Visual Arts National Assessment, by Block and Item: 2016

NAEP Technical DocumentationScore Range and Percentage of Exact Agreement for the Constructed-Response Items From Previous Year 2008, Rescored in 2016, Grade 8 Visual Arts National Assessment, by Block and Item: 2016

Score range and percentage of exact agreement for the constructed-response items from previous year 2008, rescored in 2016, grade 8 visual arts national assessment, by block and item: 2016
Block	Item	Score range	Sample size	Percentage of exact agreement
V3	VC00002	1-3	800	86
	VC00005	1-4	800	73
	VC000A4	1-3	800	80
	VC000B4	1-3	800	64
V4	VD00001	1-2	800	92
	VD00007	1-3	800	73
	VD00008	1-4	800	68
	VD00009	1-4	800	70
	VD000A5	1-3	800	73
	VD000A6	1-3	800	73
	VD000B5	1-3	800	77
	VD000B6	1-2	800	88
V5	VE00001	1-2	800	84
	VE00002	1-4	900	71
	VE00004	1-3	800	76
	VE00008	1-3	800	78
	VE00303	1-4	800	71
V6	VF00001	1-3	800	76
	VF00004	1-3	800	73
	VF000A6	1-4	800	77
	VF000A7	1-3	800	83
	VF000B6	1-4	800	75
	VF000B7	1-2	800	81
	VF000C6	1-4	800	70
	VF000C7	1-3	800	80
NOTE: Special codes assigned to student responses including omitted, off-task, and illegible were included in the calculation of the percentage of exact agreement measure. In 2016, grades 4 and 12 were not assessed. SOURCE: U.S. Department of Education, Institute of Education Sciences, National Center for Education Statistics, National Assessment of Educational Progress (NAEP), 2016 Visual Arts Assessment.

Score range, Cohen's Kappa, and intraclass correlation for the constructed-response items from previous year 2008, rescored in 2016, grade 8 visual arts national assessment, by block and item: 2016
Block	Item	Score range	Sample size	Cohen’s Kappa	Intraclass correlation
V3	VC00002	1-3	800	.79	.89
	VC00005	1-4	800	.45	.63
	VC000A4	1-3	800	.65	.82
	VC000B4	1-3	800	.37	.52
V4	VD00001	1-2	800	.82	†
	VD00007	1-3	800	.47	.60
	VD00008	1-4	700	.38	.60
	VD00009	1-4	700	.48	.66
	VD000A5	1-3	800	.54	.67
	VD000A6	1-3	800	.52	.58
	VD000B5	1-3	800	.66	.80
	VD000B6	1-2	800	.75	†
V5	VE00001	1-2	800	.65	†
	VE00002	1-4	900	.45	.60
	VE00004	1-3	700	.52	.60
	VE00008	1-3	700	.62	.78
	VE00303	1-4	800	.46	.59
V6	VF00001	1-3	800	.59	.71
	VF00004	1-3	800	.56	.68
	VF000A6	1-4	800	.65	.75
	VF000A7	1-3	800	.73	.79
	VF000B6	1-4	800	.62	.76
	VF000B7	1-2	800	.55	†
	VF000C6	1-4	800	.53	.65
	VF000C7	1-3	800	.70	.77
† Not applicable. The intraclass correlation is not reported for dichotomously scored items. NOTE: The intraclass correlation is most appropriate for items with more than two categories. Cohen's Kappa is a measure of reliability that is appropriate for both dichotomously and polytomously scored items. Special codes assigned to student responses including omitted, off-task, and illegible were not included in the calculation of Cohen's Kappa or intraclass correlation. SOURCE: U.S. Department of Education, Institute of Education Sciences, National Center for Education Statistics, National Assessment of Educational Progress (NAEP), 2016 Visual Arts Assessment.

Last updated 14 February 2022 (SK)

Printer-friendly Version