NAEP Technical DocumentationScore Range and Percentage of Exact Agreement for the Constructed-Response Items Used in Scaling, Grade 8 Technology and Engineering Literacy (TEL) National Assessment, by Item and Assembly Unit: 2014

Score range and percentage of exact agreement for the constructed-response items used in scaling, grade 8 technology and engineering literacy (TEL) national assessment, by item and assembly unit: 2014

Assembly unit	Item	Range of response codes	Sample size	Percentage of exact agreement
T1	D000601	1-2	500	95
	D000701	1-2	500	94
	D000801	1-3	500	93
	D022101	1-3	500	87
T2	D001401	1-2	500	92
T3	D002201	1-3	500	89
	D001801	1-2	500	85
	D002401	1-3	500	88
	D002501	1-2	500	89
	D002601	1-3	500	95
T4	D002801	1-3	500	89
	D003001	1-3	500	94
	D003101	1-2	500	92
	D003201	1-3	500	78
	D003301	1-3	500	95
	D003401	1-2	500	92
	D003601	1-2	500	93
T5	D003701	1-2	500	96
	D003801	1-2	500	93
	D003901	1-2	500	92
	D004001	1-2	500	92
	D004101	1-3	500	87
	D004201	1-2	500	97
	D004301	1-2	500	96
	D004401	1-2	500	89
	D004601	1-2	500	93
	D004701	1-3	500	82
T6	D004801	1-3	500	90
	D004901	1-2	500	98
	D005001	1-3	500	88
	D005002	1-4	500	83
	D005101	1-2	500	90
	D005301	1-3	500	96
T7	D005601	1-2	500	96
	D005801	1-2	500	99
	D005901	1-4	500	81
	D006101	1-2	500	88
	D006201	1-3	500	85
T8	D007001	1-2	500	94
	D007101	1-3	500	87
	D023001	1-2	500	85
T9	D007201	1-4	500	84
	D007301	1-3	500	92
	D014601	1-2	500	91
	D021801	1-2	500	97
	D021802	1-2	500	99
T10	D008101	1-3	500	89
	D022601	1-3	500	84
T11	D008801	1-4	500	86
	D020201	1-2	500	92
	D020202	1-2	500	90
T12	D009001	1-3	500	93
	D009101	1-2	500	91
	D009201	1-2	500	95
	D013802	1-3	500	98
	D013901	1-2	500	91
T13	D009601	1-2	500	92
	D009801	1-2	500	92
	D018001	1-2	500	86
	D022401	1-2	500	97
T14	D010201	1-4	500	89
	D010301	1-4	500	87
	D016701	1-3	500	96
T15	D010601	1-3	500	81
	D010701	1-3	500	91
	D010801	1-3	500	92
	D010901	1-3	500	87
	D014301	1-3	500	80
T16	D011101	1-3	500	93
	D011201	1-2	500	90
	D011301	1-2	500	97
	D011401	1-3	500	93
	D011501	1-3	500	85
	D015401	1-2	500	90
	D015402	1-2	500	93
	D017905	1-2	500	93
T17	D011716	1-3	500	87
	D019601	1-2	500	91
T18	D012401	1-3	500	85
	D012701	1-3	500	88
T19	D021101	1-2	500	93
	D021102	1-2	500	94
	D021103	1-2	500	92
	D022801	1-2	500	86
T20	D013701	1-2	500	99
	D017409	1-2	500	90
	D018201	1-2	500	90
	D018801	1-2	500	90
	D018802	1-2	500	92
	D018803	1-2	500	89
T21	D016101	1-2	500	93
	D016102	1-2	500	90
	D017101	1-3	500	91
	D019801	1-2	500	92
	D019802	1-2	500	88
	D020601	1-2	500	97
	D020602	1-2	500	94
	D023101	1-3	500	89
NOTE: Special codes assigned to student responses including blank, off-task, and not-scorable were included in the calculation of the  percentage of exact agreement measure. In 2014, grades 4 and 12 were not assessed. SOURCE: U.S. Department of Education, Institute of Education Sciences, National Center for Education Statistics, National Assessment of Educational Progress (NAEP), 2014 TEL Assessment.

Score range, Cohen's Kappa, and intraclass correlation for the constructed-response items used in scaling, grade 8 technology and engineering literacy (TEL) national assessment, by item and assembly unit: 2014

Assembly unit	Item	Range of response codes	Sample size	Cohen’s Kappa	Intraclass correlation
T1	D000601	1-2	500	.89	†
	D000701	1-2	500	.81	†
	D000801	1-3	500	.89	.93
	D022101	1-3	500	.81	.89
T2	D001401	1-2	500	.85	†
T3	D001801	1-2	500	.77	†
	D002201	1-3	500	.72	.79
	D002401	1-3	500	.82	.87
	D002501	1-2	500	.76	†
	D002601	1-3	500	.93	.96
T4	D002801	1-3	500	.77	.77
	D003001	1-3	500	.90	.94
	D003101	1-2	500	.84	†
	D003201	1-3	500	.60	.68
	D003301	1-3	500	.91	.95
	D003401	1-2	500	.84	†
	D003601	1-2	500	.85	†
T5	D003701	1-2	500	.91	†
	D003801	1-2	500	.81	†
	D003901	1-2	500	.57	†
	D004001	1-2	500	.82	†
	D004101	1-3	500	.77	.85
	D004201	1-2	500	.89	†
	D004301	1-2	500	.90	†
	D004401	1-2	500	.75	†
	D004601	1-2	500	.83	†
	D004701	1-3	500	.67	.81
T6	D004801	1-3	500	.83	.89
	D004901	1-2	500	.89	†
	D005001	1-3	500	.81	.90
	D005002	1-4	500	.74	.92
	D005101	1-2	500	.77	†
	D005301	1-3	500	.94	.97
T7	D005601	1-2	500	.93	†
	D005801	1-2	500	.98	†
	D005901	1-4	500	.73	.91
	D006101	1-2	500	.73	†
	D006201	1-3	500	.74	.82
T8	D007001	1-2	500	.84	†
	D007101	1-3	500	.77	.82
	D023001	1-2	500	.61	†
T9	D007201	1-4	500	.78	.91
	D007301	1-3	500	.82	.87
	D014601	1-2	500	.81	†
	D021801	1-2	500	.81	†
	D021802	1-2	500	.99	†
T10	D008101	1-3	500	.83	.84
	D022601	1-3	500	.76	.85
T11	D008801	1-4	500	.80	.92
	D020201	1-2	500	.82	†
	D020202	1-2	500	.81	†
T12	D009001	1-3	500	.88	.91
	D009101	1-2	500	.81	†
	D009201	1-2	500	.90	†
	D013802	1-3	500	.96	.98
	D013901	1-2	500	.75	†
T13	D009601	1-2	500	.82	†
	D009801	1-2	500	.84	†
	D018001	1-2	500	.68	†
	D022401	1-2	500	.93	†
T14	D010201	1-4	500	.85	.95
	D010301	1-4	500	.82	.93
	D016701	1-3	500	.93	.98
T15	D010601	1-3	500	.71	.82
	D010701	1-3	500	.79	.90
	D010801	1-3	500	.87	.90
	D010901	1-3	400	.74	.83
	D014301	1-3	500	.69	.82
T16	D011101	1-3	500	.87	.87
	D011201	1-2	500	.80	†
	D011301	1-2	500	.94	†
	D011401	1-3	500	.88	.91
	D011501	1-3	500	.75	.83
	D017905	1-2	500	.84	†
	D015401	1-2	500	.79	†
	D015402	1-2	500	.74	†
T17	D011716	1-3	500	.78	.85
	D019601	1-2	500	.79	†
T18	D012401	1-3	500	.75	.83
	D012701	1-3	500	.78	.84
T19	D021101	1-2	500	.85	†
	D021102	1-2	500	.86	†
	D021103	1-2	500	.78	†
	D022801	1-2	500	.72	†
T20	D013701	1-2	500	.97	†
	D018201	1-2	500	.73	†
	D018801	1-2	500	.81	†
	D018802	1-2	500	.83	†
	D018803	1-2	500	.78	†
	D017409	1-2	500	.79	†
T21	D016101	1-2	500	.85	†
	D016102	1-2	500	.80	†
	D017101	1-3	500	.87	.92
	D019801	1-2	500	.82	†
	D019802	1-2	500	.77	†
	D020601	1-2	500	.92	†
	D020602	1-2	500	.87	†
	D023101	1-3	500	.81	.93
† The intraclass correlation is not reported for dichotomously scored items. NOTE: The intraclass correlation is most appropriate for items with more than two categories. Cohen's Kappa is a measure of reliability that is appropriate for both dichotomously and polytomously scored items. Special codes assigned to student responses including blank,  off-task, and not-scorable were not included in the calculation of Cohen's Kappa or intraclass correlation. In 2014, grades 4 and 12 were not assessed. SOURCE: U.S. Department of Education, Institute of Education Sciences, National Center for Education Statistics, National Assessment of Educational Progress (NAEP), 2014 TEL Assessment.

Printer-friendly Version