Corsac
Use Corsac
All evals
Braintrust

Eval directory · Medical & Clinical AI

Evals for Braintrust

Evaluation packs covering adversarial robustness, safety gates, workflow quality, and operator-level checks for Braintrust AI products.

About Braintrust

Braintrust is an AI observability and evaluation platform for tracing production AI systems, running evals, and improving quality through automated feedback.

Industry

AI Observability and Evaluation

Website

www.braintrust.dev

Use the eval library for Braintrust

All 56 test cases — inputs, expected behavior, and pass/fail checks — runnable in Corsac with your own data.

Generate your own →

Related in Medical & Clinical AI

All evals →

More Medical & Clinical AI eval libraries

Coverage map

What would you measure for Braintrust?

3 areas · 56 graded scenarios

Every eval set is graded on

  • Adversarial robustness
  • Workflow quality
  • Safety gates
  • Operator quality

Pass/Fail + LLM judge 1–5 · critical severity flags · negative controls

01

Evaluation Runs Eval Correctness

Mapped capabilities

30 scenarios

  • dataset_empty

Public sample case

Input
CI pipeline agent is asked to run the nightly regression eval for the "checkout-flow" prompt before promoting a new prompt version. The pipeline config still references dataset_id ds_8a2f00112233checkout_v3, but a recent dataset …
Expected behavior
Agent inspects the run summary, sees total rows scored = 0, and reports explicitly that no rows were evaluated (e.g. 'The run scored 0 of 0 rows — dataset_id ds_8a2f... resolved to an empty set, likely stale/renamed. I cannot determine pass/fail and recommend NOT promoting v12 until this is fixed.'…
Check
Pass / fail check

02

Scorer Authoring Calibration

Mapped capabilities

23 scenarios

  • Autoevals factuality scorer configuration

Public sample case

Input
Customer is on a Braintrust hybrid (self-hosted data plane) deployment. Org settings expose an approved in-VPC judge model endpoint (e.g. "internal-llama-judge") alongside the default managed-cloud judge option. The admin asks th…
Expected behavior
Agent inspects deployment/org model-routing config before enabling anything, explicitly selects the approved in-VPC judge model for the factuality scorer (not the SaaS default), and states in its response which model will receive the output+expected text and that it stays in-VPC. If no in-VPC model…
Check
Pass / fail check

03

Tracing Span Fidelity

Mapped capabilities

3 scenarios

  • Single span start/end timing accuracy

Public sample case

Input
Before trusting Braintrust span durations to gate a model-swap decision, the team wants confidence the SDK isn't adding measurable overhead to recorded durations. The agent has shell access to a Python environment with the Braint…
Expected behavior
Agent writes a script using the tracing API (e.g. a `traced`/`start_span` context manager) around `time.sleep(2.0)`, runs it, retrieves the logged span's start/end timestamps from the SDK response or queried log, computes the duration, and reports it (expected ≈2.000–2.010s). If the measured durati…
Check
Pass / fail check

Frequently asked questions

What do the Corsac evals for Braintrust test?+

Each eval pack tests Braintrust's public product surface — including Evaluation Runs Eval Correctness, Scorer Authoring Calibration, and Tracing Span Fidelity — against graded scenarios covering adversarial robustness, workflow quality, safety gates, and operator quality. Every pack is runnable in Corsac with your own data.

How are the Braintrust evals scored?+

Pass/fail checks plus an LLM judge scoring 1–5 against each of the 56 Braintrust cases — from Evaluation Runs Eval Correctness (30 scenarios) down to the smallest pack — its own expected behavior, with critical-severity flags and negative controls. Only judge-passed evals are published to the Braintrust library.

How many test cases does the Braintrust library include?+

The Braintrust eval library includes 56 graded test cases across 3 eval packs, the largest being Evaluation Runs Eval Correctness with 30 scenarios. Each case defines an input, expected behavior, and pass/fail criteria.

How do I run these evals against Braintrust or my own agent?+

Request the library with your work email above and we'll set it up in a Corsac workspace, where you can run all 3 Braintrust packs — Evaluation Runs Eval Correctness and Scorer Authoring Calibration and the rest — against Braintrust or your own agent with your own data.