Table of Contents  |  Search Technical Documentation  |  References

NAEP Analysis and Scaling → Initial Activities → Constructed-Response Interrater Reliability → Range of Response Codes, Percent Agreement, and Cohen's Kappa or Intraclass Correlation for the Constructed-Response Items Used in Scaling, Grade 8 Reading Combined National and State Main Assessment, by Item and Block: 2007

NAEP Technical DocumentationRange of response codes, percent agreement, and Cohen's Kappa or intraclass correlation for the constructed-response items used in scaling, grade 8 reading combined national and state main assessment, by item and block: 2007

Block	Item	Range of response codes	Sample size	Percent agreement	Cohen’s Kappa	Intraclass correlation
— Not available. The intraclass correlation is not reported for dichotomously scored items; Cohen's Kappa is not reported for polytomously scored items. 1 New items introduced in 2006 Precalibration that were rescored in 2007. NOTE: Cohen's Kappa is a measure of reliability that is appropriate for items that are dichotomously scored. The intraclass correlation coefficient is most appropriate for items with more than two categories. SOURCE: U.S. Department of Education, Institute of Education Sciences, National Center for Education Statistics, National Assessment of Educational Progress (NAEP), 2007 Reading Assessment.
R3	R017101	1–2	1,500	89.83	0.78	—
	R017102	1–3	1,300	85.68	—	0.83
	R017104	1–3	1,400	88.92	—	0.88
	R017105	1–4	1,400	69.50	—	0.76
	R017107	1–3	1,400	84.86	—	0.84
	R017108	1–2	1,300	96.42	0.87	—
	R017110	1–3	1,300	91.73	—	0.92
R41	R055501	1–3	1,400	78.95	—	0.73
	R055801	1–3	1,400	82.60	—	0.67
	R056101	1–4	1,400	71.73	—	0.81
	R056301	1–3	1,400	75.34	—	0.66
R5	R012601	1–2	1,800	87.72	0.75	—
	R012604	1–2	1,500	90.06	0.80	—
	R012607	1–4	1,400	73.74	—	0.77
	R012612	1–2	1,400	87.48	0.74	—
R6	R013201	1–4	1,500	75.17	—	0.83
	R013203	1–2	1,500	95.58	0.63	—
	R013205	1–2	1,400	96.20	0.80	—
	R013207	1–2	1,400	87.25	0.71	—
	R013209	1–2	1,600	93.97	0.83	—
	R013211	1–2	1,400	86.69	0.73	—
	R013212	1–4	1,300	77.58	—	0.74
R71	R034601	1–3	1,400	83.54	—	0.81
	R034901	1–3	1,400	84.47	—	0.85
	R035001	1–3	1,400	85.14	—	0.84
	R035201	1–3	1,400	88.30	—	0.83
	R035301	1–4	1,400	70.40	—	0.84
R8	R053401	1–3	1,500	84.88	—	0.86
	R053402	1–2	1,500	90.55	0.54	—
	R053405	1–4	1,400	70.55	—	0.73
	R053408	1–3	1,500	84.47	—	0.83
	R053410	1–2	1,700	84.04	0.64	—
R91	R056501	1–2	1,400	94.42	0.87	—
	R056801	1–3	1,400	79.51	—	0.77
	R057101	1–3	1,400	71.26	—	0.69
	R057301	1–3	1,400	69.24	—	0.72
R10	R013402	1–2	1,700	98.06	0.96	—
	R013403	1–4	1,400	94.14	—	0.96
	R013405	1–2	1,400	92.61	0.85	—
	R013406	1–4	1,400	75.82	—	0.87
	R013407	1–2	1,400	91.90	0.84	—
	R013409	1–2	1,800	91.29	0.82	—
	R013411	1–2	1,400	85.89	0.69	—
	R013413	1–2	1,400	81.73	0.63	—
R11	R013001	1–2	1,400	88.07	0.71	—
	R013003	1–2	1,400	99.43	0.99	—
	R013004	1–4	1,400	67.58	—	0.82
	R013005	1–2	1,400	86.32	0.67	—
	R013007	1–2	1,300	95.39	0.87	—
	R013008	1–2	1,400	84.88	0.70	—
	R013009	1–2	1,400	87.55	0.67	—
	R013010	1–2	1,400	92.88	0.80	—
	R013011	1–2	1,400	85.76	0.70	—
R12	R053202	1–3	1,700	83.75	—	0.79
	R053203	1–3	1,600	76.60	—	0.72
	R053205	1–4	1,500	73.26	—	0.86
	R053207	1–3	1,500	78.35	—	0.72
	R053209	1–3	1,500	85.54	—	0.81
R13	R016201	1–3	1,300	78.50	—	0.60
	R016202	1–3	1,500	83.62	—	0.76
	R016204	1–4	1,200	79.85	—	0.77
	R016205	1–3	1,200	80.77	—	0.77
	R016207	1–3	1,200	92.60	—	0.94
	R016210	1–4	1,200	64.21	—	0.73
	R016211	1–3	1,200	83.05	—	0.81
	R016212	1–3	1,200	87.48	—	0.89
	R016213	1–3	1,200	78.90	—	0.76
R14	R026501	1–3	1,500	80.66	—	0.80
	R026801	1–4	1,400	75.37	—	0.83
	R027001	1–2	1,300	89.82	0.67	—
	R027201	1–3	1,400	86.34	—	0.84
R15	R028401	1–3	2,100	85.84	—	0.84
	R028501	1–3	2,000	86.74	—	0.86
	R028801	1–4	1,600	75.78	—	0.86
	R029601	1–3	1,400	81.55	—	0.79

Printer-friendly Version