NAEP Technical Documentation

Score range and percentage of exact agreement for the constructed-response items from the previous year 2010, rescored in 2014, grade 8 civics national assessment, by item and block: 2014

Block	Item	Range of response codes	Sample size	Percentage of exact agreement
C3	P042171	1-3	800	87
C3	P042174	1-3	800	84
C3	P042177	1-3	800	85
C3	P042181	1-3	800	83
C3	P042184	1-3	800	81
C4	P051401	1-3	800	86
C4	P052101	1-4	800	75
C4	P053801	1-3	800	80
C4	P054101	1-3	800	78
C5	P040903	1-3	800	60
C5	P040906	1-3	800	86
C5	P040910	1-3	800	77
C5	P040913	1-3	800	95
C6	P049201	1-3	800	78
C6	P049501	1-3	800	81
C6	P050501	1-4	800	84
C6	P054801	1-3	800	96
C6	P062201	1-3	800	84
C7	P041102	1-3	800	88
C7	P041106	1-4	800	72
C7	P041111	1-3	800	84
C7	P041116	1-3	800	72
C8	P041202	1-3	800	82
C8	P041205	1-3	800	78
C8	P041213	1-4	800	78
C9	P041307	1-4	800	86
C9	P041309	1-3	800	87
C9	P041315	1-3	800	84
C10	P042152	1-3	800	93
C10	P042155	1-4	800	71
C10	P042158	1-3	800	89
C10	P042162	1-3	800	77
NOTE: Special codes assigned to student responses including blank, off-task, and not-scorable were included in the calculation of the Percentage of Exact Agreement measure. In 2014, grades 4 and 12 were not assessed. SOURCE: U.S. Department of Education, Institute of Education Sciences, National Center for Education Statistics, National Assessment of Educational Progress (NAEP), 2014 Civics Assessment.

Score range, Cohen's Kappa, and intraclass correlation for the constructed-response items from the previous year 2010, rescored in 2014, grade 8 civics assessment, by item and block: 2014

Block	Item	Range of response codes	Sample size	Cohen’s Kappa	Intraclass correlation
C3	P042171	1-3	800	.79	.87
C3	P042174	1-3	700	.53	.54
C3	P042177	1-3	700	.74	.85
C3	P042181	1-3	700	.69	.79
C3	P042184	1-3	700	.65	.75
C4	P051401	1-3	700	.77	.82
C4	P052101	1-4	700	.57	.74
C4	P053801	1-3	700	.53	.63
C4	P054101	1-3	700	.60	.68
C5	P040903	1-3	800	.32	.48
C5	P040906	1-3	700	.63	.79
C5	P040910	1-3	700	.56	.63
C5	P040913	1-3	700	.90	.93
C6	P049201	1-3	800	.65	.75
C6	P049501	1-3	700	.65	.76
C6	P050501	1-4	700	.76	.89
C6	P054801	1-3	700	.92	.96
C6	P062201	1-3	700	.70	.78
C7	P041102	1-3	700	.77	.85
C7	P041106	1-4	700	.53	.63
C7	P041111	1-3	700	.65	.77
C7	P041116	1-3	700	.43	.58
C8	P041202	1-3	700	.70	.81
C8	P041205	1-3	800	.54	.66
C8	P041213	1-4	700	.64	.82
C9	P041307	1-4	700	.76	.89
C9	P041309	1-3	700	.77	.85
C9	P041315	1-3	700	.71	.79
C10	P042152	1-3	800	.89	.93
C10	P042155	1-4	700	.57	.77
C10	P042158	1-3	700	.81	.85
C10	P042162	1-3	700	.56	.70
NOTE: The intraclass correlation is most appropriate for items with more than two categories. Cohen's Kappa is a measure of reliability that is appropriate for both dichotomously and polytomously scored items. Special codes assigned to student responses including blank, off-task, and not-scorable were not included in the calculation of Cohen's Kappa or intraclass correlation. In 2014, grades 4 and 12 were not assessed.  SOURCE: U.S. Department of Education, Institute of Education Sciences, National Center for Education Statistics, National Assessment of Educational Progress (NAEP), 2014 Civics Assessment.

Printer-friendly Version