Custom Evaluators And Criteria

Operator defines a custom evaluator with the criterion 'the answer should be professional' and nothing else.

Write custom criteria as concrete, checkable statements with observable anchors (e.g. 'no profanity; addresses the user's question; cites at least one provided source when one exists') rather than abstract adjectives. Specify pass/fail boundaries so two reviewers (human or model) would agree on the…

Operator writes a custom evaluator and immediately puts it in production gating without checking it against any labeled examples.

Validate a new custom evaluator on a labeled set with known-good and known-bad examples (including edge cases) before trusting it for gating. Confirm it passes the goods and fails the bads, measure agreement with human labels, and iterate the criteria where it disagrees. Promote to gating only afte…

Operator writes a single custom evaluator that simultaneously judges factual accuracy, tone, formatting, and safety, returning one blended score.

Decompose distinct concerns into separate evaluators (accuracy, tone, format, safety) so each yields an actionable, independently-thresholded verdict. A single blended score hides which dimension failed and prevents per-concern gating policy. Compose them at the policy layer, not inside one rubric.