
Tracing And Runs Api
LangSmith · LangSmith
LLM Observability & Evaluation Platform — LangSmith (LangChain)
Evaluates LangSmith's Tracing & Runs API across 9 scenario-based test cases, each graded against an expected-behavior rubric by an LLM judge, from Corsac's LLM Observability & Evaluation Platform eval coverage.
About LangSmith
LangSmith is LangChain's LLM observability and evaluation platform: tracing, datasets, evaluators (LLM-as-judge, code, and human), experiments, prompt management, and online monitoring used by AI teams to measure and improve LLM apps in production.
Employees
~200
Industry
LLM Observability
Headquarters
San Francisco, CA
Website
www.langchain.com/langsmithSample tests· showing 3 of 9
| # | Input | Expected behavior | Check |
|---|---|---|---|
| 01 | Agent wraps a multi-step pipeline with @traceable on the orchestrator and child @traceable on each step. A trace tree is expected with the orchestrator as parent_run and three child runs under it. | Decorate the orchestrator and each step with @traceable; rely on LANGSMITH_TRACING=true and Python contextvars (or run_tree propagation) to auto-attach child runs to the active parent. Do NOT pass parent_run_id manually unless crossing a thread/process boundary — incorrect manual ids create orphane… | Pass / FailAi Platformhigh |
| 02 | An agent issues three sub-operations: an LLM call, a vector search retriever, and a tool execution. Operator wants per-type cost and latency aggregates in the LangSmith Projects view. | Set run_type explicitly on each @traceable: 'llm' for model calls, 'retriever' for retrieval, 'tool' for tool invocations, 'chain' for orchestration. LangSmith's per-type filters and cost rollups depend on run_type being correct — defaulting everything to 'chain' loses the breakdown. | Pass / FailAi Platformhigh |
| 03 | Service A starts a trace, calls service B over HTTP. Service B is expected to show as a child run under A's trace in LangSmith. | Service A serializes get_current_run_tree().to_headers() and adds them to the outbound HTTP request. Service B reads langsmith-trace and baggage headers via TracingMiddleware (or with tracing_context(parent=headers)) and resumes the trace context. Both services use the same LangSmith project. | Pass / FailAi Platformhigh |
How this eval is graded
Grade against expected.ideal_behavior and expected.rubric. Per-criterion pass requires mean >= 4.0 and no criterion below 3.
Rubric criteria
- Langsmith
- Ai Platform
- Tracing And Runs Api
Recommended for
Works with
Related evals
Claude API
Evaluates Anthropic's Batch API across 9 scenario-based test cases, each graded against an expected-behavior rubric by an LLM judge, from Corsac's Foundation Model & API eval coverage.
View AI PlatformClaude API
Evaluates Anthropic's Extended Thinking across 9 scenario-based test cases, each graded against an expected-behavior rubric by an LLM judge, from Corsac's Foundation Model & API eval coverage.
View AI PlatformClaude API
Evaluates Anthropic's Files API & Citations across 9 scenario-based test cases, each graded against an expected-behavior rubric by an LLM judge, from Corsac's Foundation Model & API eval coverage.
ViewFrequently asked questions
What does the Tracing And Runs Api eval for LangSmith LangSmith test?+
Evaluates LangSmith's Tracing & Runs API across 9 scenario-based test cases, each graded against an expected-behavior rubric by an LLM judge, from Corsac's LLM Observability & Evaluation Platform eval coverage.
How is the Tracing And Runs Api eval scored?+
The judge rubric: Grade against expected.ideal_behavior and expected.rubric. Per-criterion pass requires mean >= 4.0 and no criterion below 3.
How many test cases does this eval pack include?+
The Tracing And Runs Api pack for LangSmith LangSmith contains 9 test cases. 3 sample cases are shown free on this page; the full set runs in a Corsac workspace.
How do I run this eval?+
Sign up for Corsac, connect your model or agent endpoint, and run the Tracing And Runs Api pack as-is or after customizing thresholds. Results land in your workspace with per-case scores, and you can gate releases on the pack in CI via the REST API.
Run this eval in your workspace
Connect your data, configure thresholds, and review results with your team.