LangSmith
For LangSmithAI Platform

Tracing And Runs Api

LangSmith · LangSmith

LLM Observability & Evaluation Platform — LangSmith (LangChain)

Evaluates LangSmith's Tracing & Runs API across 9 scenario-based test cases, each graded against an expected-behavior rubric by an LLM judge, from Corsac's LLM Observability & Evaluation Platform eval coverage.

About LangSmith

LangSmith is LangChain's LLM observability and evaluation platform: tracing, datasets, evaluators (LLM-as-judge, code, and human), experiments, prompt management, and online monitoring used by AI teams to measure and improve LLM apps in production.

Employees

~200

Industry

LLM Observability

Headquarters

San Francisco, CA

Sample tests· showing 3 of 9

#InputExpected behaviorCheck
01

Agent wraps a multi-step pipeline with @traceable on the orchestrator and child @traceable on each step. A trace tree is expected with the orchestrator as parent_run and three child runs under it.

Decorate the orchestrator and each step with @traceable; rely on LANGSMITH_TRACING=true and Python contextvars (or run_tree propagation) to auto-attach child runs to the active parent. Do NOT pass parent_run_id manually unless crossing a thread/process boundary — incorrect manual ids create orphane…

Pass / FailAi Platformhigh
02

An agent issues three sub-operations: an LLM call, a vector search retriever, and a tool execution. Operator wants per-type cost and latency aggregates in the LangSmith Projects view.

Set run_type explicitly on each @traceable: 'llm' for model calls, 'retriever' for retrieval, 'tool' for tool invocations, 'chain' for orchestration. LangSmith's per-type filters and cost rollups depend on run_type being correct — defaulting everything to 'chain' loses the breakdown.

Pass / FailAi Platformhigh
03

Service A starts a trace, calls service B over HTTP. Service B is expected to show as a child run under A's trace in LangSmith.

Service A serializes get_current_run_tree().to_headers() and adds them to the outbound HTTP request. Service B reads langsmith-trace and baggage headers via TracingMiddleware (or with tracing_context(parent=headers)) and resumes the trace context. Both services use the same LangSmith project.

Pass / FailAi Platformhigh

Unlock full benchmark

6 more test cases

Use this benchmark

How this eval is graded

Grade against expected.ideal_behavior and expected.rubric. Per-criterion pass requires mean >= 4.0 and no criterion below 3.

Rubric criteria

  • Langsmith
  • Ai Platform
  • Tracing And Runs Api

Recommended for

LangSmithLangSmith customers

Works with

Related evals

Frequently asked questions

What does the Tracing And Runs Api eval for LangSmith LangSmith test?+

Evaluates LangSmith's Tracing & Runs API across 9 scenario-based test cases, each graded against an expected-behavior rubric by an LLM judge, from Corsac's LLM Observability & Evaluation Platform eval coverage.

How is the Tracing And Runs Api eval scored?+

The judge rubric: Grade against expected.ideal_behavior and expected.rubric. Per-criterion pass requires mean >= 4.0 and no criterion below 3.

How many test cases does this eval pack include?+

The Tracing And Runs Api pack for LangSmith LangSmith contains 9 test cases. 3 sample cases are shown free on this page; the full set runs in a Corsac workspace.

How do I run this eval?+

Sign up for Corsac, connect your model or agent endpoint, and run the Tracing And Runs Api pack as-is or after customizing thresholds. Results land in your workspace with per-case scores, and you can gate releases on the pack in CI via the REST API.

Run this eval in your workspace

Connect your data, configure thresholds, and review results with your team.