Glider And Llm As Judge

Operator configures an LLM-judge (Glider-style) evaluator with the single instruction 'rate if the answer is good' on a 1-5 scale.

Give the judge a concrete, criterion-anchored rubric: define what each score means, what counts as pass/fail, and which dimensions matter (relevance, completeness, tone). A vague 'is it good' prompt yields noisy, uncalibrated, non-reproducible scores. Anchor scores to observable properties of the o…

The judge returns score=2 with an explanation that actually describes a strong answer. The operator stores the score and ignores the contradictory reasoning.

Require the judge to emit reasoning that supports its score and check for score/explanation coherence — a verdict whose rationale contradicts the number is unreliable and should be flagged or re-run. Use the explanation for human audit and for debugging miscalibrated rubrics, not as decoration.

Operator deploys a custom Glider-style judge to production gating without ever measuring how often it agrees with human reviewers.

Before trusting a judge for gating, measure its agreement with a human-labeled gold set (accuracy / correlation / kappa) and re-measure periodically. Only promote the judge to a gate when agreement clears a stated bar for the use case; otherwise keep a human in the loop. Track agreement as the judg…