Quality Control And Calibration

Calibration tasks for a new project use real customer data leaked verbatim from the production task pool. Labelers see the calibration items in their actual workload.

Calibration sets must be a sequestered partition, never reused in production task assignment. Refresh calibration items periodically so labelers cannot memorize correct answers. Audit for calibration leakage into production pools and remove.

A senior reviewer's pass-rate trends from 65% to 92% over 3 months on the same task type; no underlying labeler-quality change.

Track reviewer-side drift independently from labeler quality: control-chart reviewer pass-rates against the cohort, flag drifts, rotate reviewers through calibration sets. Do not let one drifting reviewer set the de facto rubric for a project.

Customer reports defects post-delivery. Mercor's tracker logs defects as a single rolling count with no association back to the offending labeler or batch.

Defect-tracking must close the loop: each defect is associated with the labeler, batch, instruction-pack version, and reviewer chain. Aggregate to spot systemic issues (a guideline phrasing, a reviewer, a cohort) and feed the next calibration cycle.