Table of Contents | Search Technical Documentation | References

NAEP Analysis and Scaling → Initial Activities → Constructed-Response Interrater Reliability → Score Range and Percentage of Exact Agreement for the Constructed-Response Items Used in Scaling, Grade 4 Science Combined National and State Assessment, by Item and Block: 2015

NAEP Technical DocumentationScore Range and Percentage of Exact Agreement for the Constructed-Response Items Used in Scaling, Grade 4 Science Combined National and State Assessment, by Item and Block: 2015

Score range and percentage of exact agreement for the constructed-response items used in scaling, grade 4 science combined national and state assessment, by item and block: 2015
Block	Item	Score range	Sample size	Percentage of exact agreement
S3	K077602	1-4	1,400	86
	K080404	1-3	1,400	87
	K153001	1-3	1,400	85
	K153102	1-4	1,400	93
	K106701	1-3	1,400	90
S4	K078902	1-3	1,300	91
	K084301	1-2	1,300	89
	K084302	1-2	1,300	90
	K153301	1-3	1,300	89
	K079501	1-4	1,300	90
S5	K081601	1-3	1,300	94
	K087301	1-3	1,300	91
	K081801	1-3	1,300	88
	K082501	1-3	1,300	96
	K084202	1-3	1,300	95
S6	K107902	1-3	1,300	87
	K086901	1-2	1,300	99
	K086902	1-2	1,300	99
	K086903	1-2	1,300	99
	K086904	1-2	1,300	99
	K076501	1-3	1,300	90
	K087402	1-4	1,300	81
	K109102	1-3	1,300	81
S7	K164901	1-3	1,300	88
	K159701	1-3	1,300	86
	K159702	1-3	1,300	88
S8	K077501	1-3	1,300	90
	K077701	1-3	1,300	96
	K078102	1-4	1,300	92
	K109701	1-3	1,300	91
	K109702	1-3	1,300	94
	K109703	1-3	1,300	93
	K078301	1-3	1,300	95
S9	K088302	1-3	1,300	98
	K088401	1-5	1,300	80
	K109402	1-3	1,300	99
	K153701	1-3	1,300	93
	K153702	1-3	1,300	93
S10	K082901	1-3	1,300	93
	K082902	1-3	1,300	93
	K082903	1-3	1,300	94
	K154001	1-3	1,300	86
	K083502	1-3	1,300	87
	K085302	1-3	1,300	90
	K083402	1-4	1,300	90
S11	K163301	1-2	1,300	97
	K163302	1-2	1,300	93
	K163303	1-2	1,300	93
NOTE: Special codes assigned to student responses including blank, off-task, and not-scorable were included in the calculation of the percentage of exact agreement measure. SOURCE: U.S. Department of Education, Institute of Education Sciences, National Center for Education Statistics, National Assessment of Educational Progress (NAEP), 2015 Science Assessment.

Score range, Cohen's Kappa, and intraclass correlation for the constructed-response items used in scaling, grade 4 science combined national and state assessment, by item and block: 2015
Block	Item	Score range	Sample size	Cohen’s Kappa	Intraclass correlation
S3	K077602	1-4	1,400	.84	.94
	K080404	1-3	1,400	.80	.86
	K153001	1-3	1,200	.72	.78
	K153102	1-4	1,100	.88	.95
	K106701	1-3	1,000	.81	.87
S4	K078902	1-3	1,300	.85	.86
	K084301	1-2	1,300	.82	†
	K084302	1-2	1,200	.84	†
	K153301	1-3	1,200	.85	.91
	K079501	1-4	1,100	.85	.92
S5	K081601	1-3	1,300	.80	.87
	K087301	1-3	1,300	.85	.90
	K081801	1-3	1,200	.80	.87
	K082501	1-3	1,100	.92	.95
	K084202	1-3	1,000	.74	.82
S6	K107902	1-3	1,300	.80	.87
	K086901	1-2	1,200	.99	†
	K086902	1-2	1,200	.99	†
	K086903	1-2	1,200	.99	†
	K086904	1-2	1,200	.99	†
	K076501	1-3	1,200	.86	.92
	K087402	1-4	1,200	.72	.81
	K109102	1-3	1,100	.58	.67
S7	K164901	1-3	1,300	.77	.82
	K159701	1-3	1,300	.77	.85
	K159702	1-3	1,300	.81	.89
S8	K077501	1-3	1,200	.82	.88
	K077701	1-3	1,300	.93	.97
	K078102	1-4	1,200	.86	.94
	K109701	1-3	1,000	.76	.80
	K109702	1-3	1,000	.80	.82
	K109703	1-3	900	.82	.84
	K078301	1-3	900	.89	.93
S9	K088302	1-3	1,300	.96	.97
	K088401	1-5	1,200	.72	.89
	K109402	1-3	1,100	.96	.99
	K153701	1-3	1,100	.85	.91
	K153702	1-3	1,100	.86	.88
S10	K082901	1-3	1,200	.85	.88
	K082902	1-3	1,200	.87	.90
	K082903	1-3	1,200	.88	.91
	K154001	1-3	1,300	.73	.82
	K083502	1-3	1,200	.75	.82
	K085302	1-3	1,100	.81	.85
	K083402	1-4	1,000	.82	.91
S11	K163301	1-2	1,200	.94	†
	K163302	1-2	1,200	.83	†
	K163303	1-2	1,200	.80	†
† Not applicable. The intraclass correlation is not reported for dichotomously scored items. NOTE: The intraclass correlation is most appropriate for items with more than two categories. Cohen's Kappa is a measure of reliability that is appropriate for both dichotomously and polytomously scored items. Special codes assigned to student responses including blank, off-task, and not-scorable were not included in the calculation of Cohen's Kappa or intraclass correlation. SOURCE: U.S. Department of Education, Institute of Education Sciences, National Center for Education Statistics, National Assessment of Educational Progress (NAEP), 2015 Science Assessment.

Last updated 15 November 2021 (ML)

Printer-friendly Version