
Agents Langgraph
LangChain (+ LangGraph) · LangChain
LLM Orchestration Framework — LangChain
Evaluates LangChain's Agents (LangGraph) across 9 scenario-based test cases, each graded against an expected-behavior rubric by an LLM judge, from Corsac's LLM Orchestration Framework eval coverage.
About LangChain
LangChain is the open-source framework for building LLM applications and agents — provider-agnostic chat-model abstractions, LCEL/Runnables composition, tools, retrieval, and the LangGraph agent runtime (Python & JS). The company also offers LangSmith (observability) and LangGraph Platform.
Sample tests· showing 3 of 9
| # | Input | Expected behavior | Check |
|---|---|---|---|
| 01 | Integrator wants a tool-calling agent and hand-builds a StateGraph from scratch, reimplementing the model/tool loop and introducing routing bugs. | Use langgraph.prebuilt.create_react_agent(model, tools) for the standard ReAct tool-calling loop; it wires the model node, ToolNode, and conditional routing back to the model. Drop to a custom StateGraph only when the prebuilt loop is insufficient. | Pass / FailAi Platformhigh |
| 02 | An agent loops between two tools forever; the integrator has no recursion_limit set and the run consumes tokens until it is killed manually. | Set a sensible recursion_limit in the config so LangGraph raises GraphRecursionError when the step budget is exceeded, then handle it (surface to the user / inspect state). Do not raise the limit blindly to 'make it finish' — investigate the loop. | Pass / FailAi Platformcritical |
| 03 | A multi-turn agent forgets prior turns because the integrator invokes it without a checkpointer or a stable thread_id. | Compile the agent with a checkpointer (MemorySaver for dev, Sqlite/Postgres for prod) and pass config={'configurable': {'thread_id': ...}} per conversation. State for that thread is restored across invocations; a missing thread_id means no continuity. | Pass / FailAi Platformhigh |
How this eval is graded
Grade against expected.ideal_behavior and expected.rubric. Per-criterion pass requires mean >= 4.0 and no criterion below 3.
Rubric criteria
- Langchain
- Ai Platform
- Agents Langgraph
Recommended for
Works with
Related evals
Claude API
Evaluates Anthropic's Batch API across 9 scenario-based test cases, each graded against an expected-behavior rubric by an LLM judge, from Corsac's Foundation Model & API eval coverage.
View AI PlatformClaude API
Evaluates Anthropic's Extended Thinking across 9 scenario-based test cases, each graded against an expected-behavior rubric by an LLM judge, from Corsac's Foundation Model & API eval coverage.
View AI PlatformClaude API
Evaluates Anthropic's Files API & Citations across 9 scenario-based test cases, each graded against an expected-behavior rubric by an LLM judge, from Corsac's Foundation Model & API eval coverage.
ViewFrequently asked questions
What does the Agents Langgraph eval for LangChain LangChain (+ LangGraph) test?+
Evaluates LangChain's Agents (LangGraph) across 9 scenario-based test cases, each graded against an expected-behavior rubric by an LLM judge, from Corsac's LLM Orchestration Framework eval coverage.
How is the Agents Langgraph eval scored?+
The judge rubric: Grade against expected.ideal_behavior and expected.rubric. Per-criterion pass requires mean >= 4.0 and no criterion below 3.
How many test cases does this eval pack include?+
The Agents Langgraph pack for LangChain LangChain (+ LangGraph) contains 9 test cases. 3 sample cases are shown free on this page; the full set runs in a Corsac workspace.
How do I run this eval?+
Sign up for Corsac, connect your model or agent endpoint, and run the Agents Langgraph pack as-is or after customizing thresholds. Results land in your workspace with per-case scores, and you can gate releases on the pack in CI via the REST API.
Run this eval in your workspace
Connect your data, configure thresholds, and review results with your team.