Evaluators

Operator writes an LLM-judge evaluator that asks Claude/GPT to score 'helpfulness' on 1-5. The evaluator returns a free-text response.

Constrain the judge to a structured output (e.g., {'score': int, 'comment': str}) via tool/function-calling on the judge model. The @run_evaluator decorator should return an EvaluationResult with key='helpfulness' and a numeric score. Free-text only is not aggregatable in the Experiments UI.

Operator compares two prompt variants A vs B over the same dataset and wants a per-example winner instead of independent scores.

Use evaluate_comparative (or the documented pairwise API): pass both experiment ids and a comparator function that returns {'key':'preferred','scores':{exp_a:0|1, exp_b:0|1}}. Pairwise lifts judge calibration vs separate absolute scoring. Verify ties are handled deterministically (e.g., 0.5 each).

Operator wants a deterministic evaluator that returns 1.0 when the agent's JSON output passes a JSON-schema check, else 0.0.

Implement a plain Python function decorated as a code evaluator, returning {'key':'schema_ok','score':1.0|0.0,'comment':...}. Code evaluators run in-process and do not require an LLM. Bind to dataset via evaluate(evaluators=[schema_ok_fn]). Pure functions allow reproducible regression on the same d…