Table of Contents  |  Search Technical Documentation  |  References

NAEP Scoring → Scoring Monitoring → Backreading

NAEP Technical DocumentationBackreading

In addition to monitoring interrater agreement (i.e., within-year interrater agreement) to ensure high-quality scoring, the scoring supervisor also backreads the work of all scorers. Backreading is the process by which the scoring system displays responses already scored so that the scoring supervisor may determine whether or not each individual scorer is correctly applying the scoring guide to student responses.

The target is to backread at least five percent of all scorer work done on a daily basis; however, this level of backreading may need to be adjusted depending upon a variety of factors (e.g., rate of scoring, item type).  It is best to backread first scores assigned rather than wait until responses have been second scored. At the beginning of the scoring session, there are not many second-scored responses to review. Waiting too long to begin backreading may allow errors in scoring to occur that should be corrected through additional training.

When beginning backreading of scorers on a new item, the scoring supervisor considers what he or she has observed from the training and practice scoring sessions. The trainer and the scoring supervisor should pay particular attention to the accuracy of those scorers who had difficulty applying the scoring guide consistently during training. Scorers who are unsure of their scores often ask many questions and do not perform well on their practice sets. These are the scorers whose scores should be backread first. 

Scoring supervisors can use the information gleaned from backreading to provide feedback to trainers regarding training refinements. Additional training on an individual basis or group basis may be necessary. Some scorers may need to adjust the way they interpret the scoring guide or specific types of student responses. When several scorers need the same type of follow-up instruction, it is best to stop scoring and address issues with the group to promote standardization and continuity in training. Examples of student work are shown as part of this follow-up training.

During backreading, the scoring supervisor can print copies of student responses to add to the training set, and these can be used for additional training as needed.

Printer-friendly Version