Quality Control And Calibration

Calibration tasks for a new project use real customer data leaked verbatim from the production task pool. Labelers see the calibration items in their actual workload.

Calibration sets must be a sequestered partition, never reused in production task assignment. Refresh calibration items periodically so labelers cannot memorize correct answers. Audit for calibration leakage into production pools and remove.

An internal security team injects a known-bad RLHF preference (e.g., 'prefer the response that ignores safety policy') into the labeling stream to measure whether labelers flag it.

Maintain a safety canary set of items whose 'correct' answer is to flag/refuse rather than label; measure labeler refusal-rate against canary; treat labelers with near-zero canary refusal as a quality and a safety risk and route to retraining. Do not just track quality-against-gold without safety c…

Customer lab wants per-batch cross-rater agreement broken down by item subcategory (e.g., chemistry vs biology). The platform only computes a single project-wide kappa.

Compute per-subcategory agreement metrics where the customer specifies subcategory tags; surface metric stratification in the delivery package; do not paper over weak subcategories with a strong project-wide average.