Table of Contents | Search Technical Documentation | References

NAEP Analysis and Scaling → Initial Activities → Constructed-Response Interrater Reliability → Score Range and Percentage of Exact Agreement for the Constructed-Response Items Used in Scaling, Grade 8 Music National Assessment, by Block and Item: 2016

NAEP Technical DocumentationScore Range and Percentage of Exact Agreement for the Constructed-Response Items Used in Scaling, Grade 8 Music National Assessment, by Block and Item: 2016

Score range and percentage of exact agreement for the constructed-response items used in scaling, grade 8 music national assessment, by block and item: 2016
Block	Item	Score range	Sample size	Percentage of exact agreement
U3	UC00004	1-2	500	100
	UC00006	1-3	500	88
	UC00010	1-4	500	86
	UC00012	1-2	500	98
	UC000A2	1-2	500	95
	UC000A3	1-3	500	90
	UC000B1	1-3	500	95
	UC000B2	1-2	500	97
	UC000B3	1-3	500	87
	UC00B14	1-3	500	91
	UC00B15	1-3	500	87
	UC00C14	1-3	500	88
	UC00C15	1-3	500	90
	UC00D15	1-3	500	88
U4	UD00007	1-3	500	88
	UD000A5	1-2	500	99
	UD000A6	1-3	500	91
	UD000B5	1-2	500	96
	UD000B6	1-3	500	82
	UD000B9	1-3	500	95
	UD000C5	1-2	500	86
	UD000C6	1-2	500	88
	UD000D5	1-2	500	95
	UD000D6	1-3	500	94
	UD000E5	1-3	500	87
	UD000E6	1-3	500	94
U5	UE00004	1-4	600	95
	UE00010	1-3	600	83
	UE00015	1-3	600	85
	UE00016	1-3	600	76
	UE000A6	1-2	600	98
	UE000A8	1-3	600	95
	UE000A9	1-3	600	86
	UE000B6	1-2	600	98
	UE000B8	1-3	600	94
	UE000B9	1-3	600	81
	UE000C8	1-2	600	97
	UE000C9	1-3	600	88
	UE00B11	1-3	600	90
	UE00B12	1-3	600	93
U6	UF00003	1-4	500	76
	UF00005	1-5	500	89
	UF000B1	1-3	500	90
	UF000B2	1-3	500	96
	UF000H1	1-3	600	93
	UF000L1	1-3	500	89
	UF000N1	1-3	500	96
	UF000P1	1-3	500	92
	UF000T1	1-3	500	92
	UF001A6	1-3	500	95
	UF001B6	1-4	500	88
	UF001C6	1-3	500	86
	UF002A6	1-3	500	96
	UF002B6	1-4	500	85
	UF002C6	1-3	500	82
	UF003A6	1-3	500	94
	UF003B6	1-4	500	84
	UF003C6	1-3	500	78
	UF004A6	1-3	500	96
	UF004B6	1-4	500	86
	UF004C6	1-3	500	85
NOTE: Special codes assigned to student responses including omitted, off-task, and illegible were included in the calculation of the percentage of exact agreement measure. SOURCE: U.S. Department of Education, Institute of Education Sciences, National Center for Education Statistics, National Assessment of Educational Progress (NAEP), 2016 Music Assessment.

Score range, Cohen's Kappa, and intraclass correlation for the constructed-response items used in scaling, grade 8 music national assessment, by block and item: 2016
Block	Item	Score range	Sample size	Cohen’s Kappa	Intraclass correlation
U3	UC00004	1-2	500	.99	†
	UC00006	1-3	500	.75	.78
	UC00010	1-4	400	.71	.82
	UC00012	1-2	400	.93	†
	UC000A2	1-2	500	.88	†
	UC000A3	1-3	500	.78	.84
	UC000B1	1-3	500	.67	.67
	UC000B2	1-2	200	1.00	†
	UC000B3	1-3	500	.72	.76
	UC00B14	1-3	500	.80	.79
	UC00B15	1-3	500	.73	.74
	UC00C14	1-3	500	.75	.75
	UC00C15	1-3	500	.79	.82
	UC00D15	1-3	500	.69	.76
U4	UD00007	1-3	500	.71	.80
	UD000A5	1-2	500	.95	†
	UD000A6	1-3	500	.86	.91
	UD000B5	1-2	500	.92	†
	UD000B6	1-3	500	.72	.82
	UD000B9	1-3	500	.92	.93
	UD000C5	1-2	500	.69	†
	UD000C6	1-2	500	.73	†
	UD000D5	1-2	500	.88	†
	UD000D6	1-3	500	.84	.90
	UD000E5	1-3	500	.54	.66
	UD000E6	1-3	500	.84	.84
U5	UE00004	1-4	500	.95	.98
	UE00010	1-3	600	.70	.81
	UE00015	1-3	500	.71	.74
	UE00016	1-3	500	.60	.75
	UE000A6	1-2	400	.96	†
	UE000A8	1-3	500	.90	.94
	UE000A9	1-3	500	.78	.89
	UE000B6	1-2	300	.94	†
	UE000B8	1-3	500	.87	.92
	UE000B9	1-3	500	.69	.80
	UE000C8	1-2	500	.93	†
	UE000C9	1-3	500	.77	.88
	UE00B11	1-3	600	.83	.85
	UE00B12	1-3	500	.86	.92
U6	UF00003	1-4	500	.65	.83
	UF00005	1-5	500	.67	.77
	UF000B1	1-3	400	.75	.86
	UF000B2	1-3	500	.70	.75
	UF000H1	1-3	300	.83	.88
	UF000L1	1-3	300	.70	.80
	UF000N1	1-3	400	.94	.96
	UF000P1	1-3	200	.51	.63
	UF000T1	1-3	200	.70	.78
	UF001A6	1-3	500	.92	.94
	UF001B6	1-4	500	.79	.88
	UF001C6	1-3	500	.72	.78
	UF002A6	1-3	500	.91	.92
	UF002B6	1-4	500	.73	.82
	UF002C6	1-3	400	.55	.58
	UF003A6	1-3	500	.87	.90
	UF003B6	1-4	500	.69	.80
	UF003C6	1-3	500	.46	.60
	UF004A6	1-3	500	.82	.84
	UF004B6	1-4	500	.75	.82
	UF004C6	1-3	500	.70	.80
† Not applicable. The intraclass correlation is not reported for dichotomously scored items. NOTE: The intraclass correlation is most appropriate for items with more than two categories. Cohen's Kappa is a measure of reliability that is appropriate for both dichotomously and polytomously scored items. Special codes assigned to student responses including omitted, off-task, and illegible were not included in the calculation of Cohen's Kappa or intraclass correlation. SOURCE: U.S. Department of Education, Institute of Education Sciences, National Center for Education Statistics, National Assessment of Educational Progress (NAEP), 2016 Music Assessment.

Last updated 14 February 2022 (SK)

Printer-friendly Version