Skip to main content
IES    
NCES
National Center for
Education Statistics
Menu
Contact
Tools
ED.gov
Newsflash
Search IES

Table of Contents | Search Technical Documentation | References

NAEP Analysis and Scaling → Initial Activities → Constructed-Response Interrater Reliability → Score Range and Percent Agreement for the Constructed-Response Items Used in Scaling, Grade 8 Reading Combined National and State Assessment, by Item and Block: 2013
NAEP Technical DocumentationScore range and percent agreement for the constructed-response items used in scaling, grade 8 reading combined national and state assessment, by item and block: 2013
Block Item Range of response codes Sample size 1  Percent exact agreement
1 Blocks R1, R3, R6, R10, R11, and R12 have larger sample sizes due to their pairing with integrated pilot blocks.
NOTE: Special codes assigned to student responses including blank, off-task, and not-scorable were included in the calculation of the Percent Exact Agreement measure.
SOURCE: U.S. Department of Education, Institute of Education Sciences, National Center for Education Statistics, National Assessment of Educational Progress (NAEP), 2013 Reading Assessment.
R1 R059604 1-2 25,800 .91
R1 R059605 1-3 25,800 .82
R1 R059606 1-3 25,800 .82
R1 R059607 1-3 25,800 .91
R1 R059610 1-4 25,800 .83
R2 R069801 1-3 23,700 .92
R2 R069804 1-3 23,700 .83
R2 R069806 1-4 23,700 .76
R2 R069810 1-3 23,700 .82
R3 R060103 1-2 25,800 .86
R3 R060105 1-2 25,800 .93
R3 R060108 1-4 25,800 .75
R4 R060301 1-2 23,800 .87
R4 R060305 1-3 23,800 .85
R4 R060306 1-4 23,800 .75
R4 R060309 1-3 23,800 .82
R5 R058504 1-3 23,900 .80
R5 R058508 1-4 23,900 .76
R5 R058509 1-2 23,900 .92
R6 R058602 1-3 25,900 .83
R6 R058606 1-3 25,900 .87
R6 R058608 1-4 25,900 .77
R7 R058805 1-3 23,700 .85
R7 R058807 1-4 23,700 .79
R7 R058810 1-3 23,700 .83
R8 R059106 1-4 23,800 .69
R8 R059108 1-3 23,800 .84
R8 R059109 1-3 23,800 .75
R8 R059110 1-2 23,800 .93
R9 R060503 1-3 23,700 .94
R9 R060505 1-4 23,700 .84
R9 R060507 1-3 23,700 .76
R9 R060508 1-3 23,700 .80
R9 R060509 1-3 23,700 .85
R10 R060603 1-3 25,200 .84
R10 R060605 1-3 25,200 .87
R10 R060606 1-4 25,200 .79
R10 R060607 1-3 25,200 .82
R10 R060608 1-3 25,200 .90
R11 R069603 1-3 25,200 .93
R11 R069606 1-3 25,200 .83
R11 R069607 1-4 25,200 .78
R11 R069609 1-3 25,200 .89
R12 R061003 1-3 25,900 .85
R12 R061004 1-2 25,900 .96
R12 R061006 1-2 25,900 .91
R12 R061007 1-4 25,900 .76
R12 R061010 1-3 25,900 .81
R13 R069705 1-2 23,600 .92
R13 R069706 1-3 23,600 .89
R13 R069707 1-4 23,600 .79
R13 R069708 1-3 23,600 .77

 

Score range and Cohen's Kappa or intraclass correlation for the constructed-response items used in scaling, grade 8 reading combined national and state assessment, by block and item: 2013
Block Item Range of response codes Sample size 1  Cohen’s Kappa Intraclass correlation

1 Blocks R1, R3, R6, R10, R11, and R12 have larger sample sizes due to their pairing with integrated pilot blocks.
† The Intraclass correlation is not reported for dichotomously scored items.
NOTE: The intraclass correlation is most appropriate for items with more than two categories. Cohen’s Kappa is a measure of reliability that is appropriate for dichotomously and polytomously scored items. Special codes assigned to student responses including blank, off-task, and not-scorable were not included in the calculation of Cohen's Kappa or intraclass correlation. 
SOURCE: U.S. Department of Education, Institute of Education Sciences, National Center for Education Statistics, National Assessment of Educational Progress (NAEP), 2013 Reading Assessment.
R1 R059604 1-2 25,300 .68 †
R1 R059605 1-3 22,300 .66 .73
R1 R059606 1-3 23,900 .69 .81
R1 R059607 1-3 21,300 .81 .90
R1 R059610 1-4 23,600 .74 .88
R2 R069801 1-3 23,300 .82 .85
R2 R069804 1-3 22,700 .71 .81
R2 R069806 1-4 22,100 .61 .76
R2 R069810 1-3 21,300 .70 .81
R3 R060103 1-2 25,400 .69 †
R3 R060105 1-2 25,000 .84 †
R3 R060108 1-4 23,800 .63 .81
R4 R060301 1-2 23,600 .65 †
R4 R060305 1-3 23,400 .68 .71
R4 R060306 1-4 23,000 .63 .72
R4 R060309 1-3 22,200 .70 .76
R5 R058504 1-3 23,000 .64 .71
R5 R058508 1-4 22,700 .59 .76
R5 R058509 1-2 21,900 .83 †
R6 R058602 1-3 25,600 .72 .75
R6 R058606 1-3 25,100 .80 .84
R6 R058608 1-4 24,500 .66 .84
R7 R058805 1-3 23,300 .73 .79
R7 R058807 1-4 22,900 .65 .79
R7 R058810 1-3 22,200 .70 .78
R8 R059106 1-4 22,100 .52 .73
R8 R059108 1-3 22,600 .75 .86
R8 R059109 1-3 23,500 .62 .76
R8 R059110 1-2 22,100 .82 †
R9 R060503 1-3 23,300 .90 .94
R9 R060505 1-4 22,200 .75 .89
R9 R060507 1-3 23,000 .55 .64
R9 R060508 1-3 22,000 .64 .71
R9 R060509 1-3 21,200 .74 .82
R10 R060603 1-3 23,300 .73 .83
R10 R060605 1-3 24,700 .76 .80
R10 R060606 1-4 24,100 .62 .73
R10 R060607 1-3 24,800 .62 .68
R10 R060608 1-3 23,900 .84 .89
R11 R069603 1-3 23,900 .88 .92
R11 R069606 1-3 24,500 .65 .70
R11 R069607 1-4 22,900 .65 .81
R11 R069609 1-3 22,300 .81 .90
R12 R061003 1-3 25,100 .76 .85
R12 R061004 1-2 25,200 .81 †
R12 R061006 1-2 24,200 .80 †
R12 R061007 1-4 21,500 .56 .70
R12 R061010 1-3 20,200 .60 .70
R13 R069705 1-2 23,300 .79 †
R13 R069706 1-3 22,900 .81 .87
R13 R069707 1-4 21,300 .64 .83
R13 R069708 1-3 20,900 .61 .74
Last updated 21 May 2017 (DC)
IES    
NCES
National Center for
Education Statistics
Explore the Institute of Education Sciences