Table of Contents  |  Search Technical Documentation  |  References

NAEP Scoring → Training for Scoring of NAEP Items → Training the Scorers → Training Packets → Qualifying Sets

NAEP Technical DocumentationQualifying Sets

Scorers must qualify individually before they begin scoring any extended constructed-response items and for selected short constructed-response items depending on the level of complexity of the item and scoring guide.  For such items, two sets of 10 to 12 papers are prepared that cover

• the range of scores on the scoring guide,
• different levels of responses within a score for some content areas (e.g., high “2” and low “2”), and
• examples of common student responses 

The papers are randomly ordered within each set and identified with different numbering schemes between the first and second sets.

To qualify, scorers are required to achieve a certain percentage of exact agreement with the scores that were assigned to all responses in the qualifying sets. The general qualifying criteria applied for specific score-point levels are

• 90 percent exact agreement for 2-point items,
• 80 percent exact agreement for 3-point items,
• 70 percent exact agreement for 4-point items,
• 70 percent exact agreement for 5-point items, and
• 70 percent exact agreement for 6-point items.

Upon completing the qualifying sets and confirming that scorers have been deemed qualified to score, scoring begins. If a scorer does not qualify, the trainer can (a) allow the scorer to score on a probationary status, with articulated parameters (e.g., agreement, backreading and/or monitoring), (b) provide opportunities for additional training, or (c) request that the scoring content lead remove the scorer from the team for this item or for the project.

Printer-friendly Version