Evaluators

Operator writes an LLM-judge evaluator that asks Claude/GPT to score 'helpfulness' on 1-5. The evaluator returns a free-text response.

Constrain the judge to a structured output (e.g., {'score': int, 'comment': str}) via tool/function-calling on the judge model. The @run_evaluator decorator should return an EvaluationResult with key='helpfulness' and a numeric score. Free-text only is not aggregatable in the Experiments UI.

Operator wants a deterministic evaluator that returns 1.0 when the agent's JSON output passes a JSON-schema check, else 0.0.

Implement a plain Python function decorated as a code evaluator, returning {'key':'schema_ok','score':1.0|0.0,'comment':...}. Code evaluators run in-process and do not require an LLM. Bind to dataset via evaluate(evaluators=[schema_ok_fn]). Pure functions allow reproducible regression on the same d…

Reviewers score 200 production runs via the Annotation Queue UI. Scores should feed back into the same run's metrics.

Human feedback flows through client.create_feedback with run_id, key, score, and optional comment. LangSmith merges human feedback with automated evaluator scores on the run's feedback panel. Use a consistent key namespace (e.g., 'human:correctness') so human vs LLM feedback aggregate separately.