Experiments And Datasets

Operator runs an 'experiment' comparing prompt v1 and v2 but does not record which model version, evaluator, or dataset snapshot each run used.

An experiment must capture the full triple — model/prompt version, evaluator (and its version), and the exact dataset snapshot — so results are reproducible and comparisons are apples-to-apples. Vary one factor at a time; record the rest. A result you cannot reproduce is not a result.

Operator builds an evaluation dataset by copying raw production transcripts containing customer names, emails, and account numbers, then shares it broadly.

Redact or synthesize PII before a production sample becomes an evaluation dataset, especially one shared across a team or vendor. Apply the redaction policy at ingestion, track consent/retention for any real data, and prefer synthetic identities where fidelity allows. Treat the dataset as subject t…

Operator upgrades the evaluator mid-quarter and then compares this quarter's average score to last quarter's, attributing the change to model improvements.

Hold the evaluator fixed when measuring model/prompt progress over time; if the evaluator must change, re-score the historical baseline with the new evaluator before comparing. Never attribute an evaluator-driven score shift to the model. Annotate the trend line at the evaluator-change boundary.