NAEP Technical DocumentationProcessing of Student Booklets and Questionnaires

Receiving Student Booklets and Questionnaires

There are several stages of work involved in receiving and processing the documents used in NAEP paper-based (PBA) and digitally based (DBA) assessments. 

Starting in 2017, the DBA response data and questionnaires (received through the online questionnaire system) have been securely transferred to the materials distribution technology staff. The materials distribution technology team processes the data, validates that the responses are provided as expected, and converts the responses into a more easily consumable format for the design analysis reporting staff. 

The NAEP materials distribution staff creates a set of predetermined rules and specifications for its processing departments to follow the PBA session materials. Materials distribution staff perform a variety of procedures on materials received from the data collection administrators before releasing these materials into their processing system. Control systems are used to monitor all NAEP materials returned from the field. The NAEP Process Control System (PCS) contains the status of sampled schools for all sessions and their scheduled assessment dates. As materials are returned, the PCS is updated to indicate receipt dates and to document any problems discovered in the shipments. As documents are processed, the system is updated to reflect processed counts. The materials report programs are utilized to allow NAEP materials distribution staff to monitor the progress of the receipt control operations.

An alert process is used to record, monitor, and categorize all discrepant or problematic situations. Throughout the processing cycle, alert situations are either flagged by computer programs or identified during clerical check-in procedures.

Certain alerts, such as changes to administration codes on the administration form, are resolved by staff changing information due to notes written on booklet covers within the data collection contractor's electronic preassessment system. These alerts, known as "Information Alerts," are recorded directly into the PCS system, thereby eliminating the need for paper documentation. Since these problem situations are categorized and tallied as they are entered in the PCS system, project staff are able to provide timely reporting on clerical-type errors made during test administration.

In cases where an alert situation cannot be resolved by the staff who first receive the materials, the alert form is forwarded to project personnel for resolution. An example of this type of alert would be a school that returned a shipment with no administration codes on either the booklet covers or the administration form. Once resolved, the problems and resolutions are recorded online by materials distribution staff in the PCS system.

The Work Flow Management System (WFM) created for NAEP is used to track batches of student booklets through each processing step allowing the materials distribution staff to monitor the status of all work in progress. WFM is also used to analyze the current workload, by project, across all workstations. Through consistent monitoring of these data, materials distribution staff is able to assign priorities to various components of the work and to monitor all phases of the data receipt and processing.

After the PBA student booklets and questionnaires are processed, they are stored securely in the materials distribution contractor's warehouse. These are stored until approval is given by NCES to securely destroy them. The electronic data (multiple-choice, open-ended responses, and human scores) captured by the materials distribution contractor from the PBA and DBA assessments is stored on a secure server until given approval by NCES to be securely destroyed.

Printer-friendly Version