LangChain
For LangChainAI Platform

Agents Langgraph

LangChain (+ LangGraph) · LangChain

LLM Orchestration Framework — LangChain

Evaluates LangChain's Agents (LangGraph) across 9 scenario-based test cases, each graded against an expected-behavior rubric by an LLM judge, from Corsac's LLM Orchestration Framework eval coverage.

About LangChain

LangChain is the open-source framework for building LLM applications and agents — provider-agnostic chat-model abstractions, LCEL/Runnables composition, tools, retrieval, and the LangGraph agent runtime (Python & JS). The company also offers LangSmith (observability) and LangGraph Platform.

Employees

~200

Industry

Agent Framework

Headquarters

San Francisco, CA

Sample tests· showing 3 of 9

#InputExpected behaviorCheck
01

Integrator wants a tool-calling agent and hand-builds a StateGraph from scratch, reimplementing the model/tool loop and introducing routing bugs.

Use langgraph.prebuilt.create_react_agent(model, tools) for the standard ReAct tool-calling loop; it wires the model node, ToolNode, and conditional routing back to the model. Drop to a custom StateGraph only when the prebuilt loop is insufficient.

Pass / FailAi Platformhigh
02

An agent loops between two tools forever; the integrator has no recursion_limit set and the run consumes tokens until it is killed manually.

Set a sensible recursion_limit in the config so LangGraph raises GraphRecursionError when the step budget is exceeded, then handle it (surface to the user / inspect state). Do not raise the limit blindly to 'make it finish' — investigate the loop.

Pass / FailAi Platformcritical
03

A multi-turn agent forgets prior turns because the integrator invokes it without a checkpointer or a stable thread_id.

Compile the agent with a checkpointer (MemorySaver for dev, Sqlite/Postgres for prod) and pass config={'configurable': {'thread_id': ...}} per conversation. State for that thread is restored across invocations; a missing thread_id means no continuity.

Pass / FailAi Platformhigh

Unlock full benchmark

6 more test cases

Use this benchmark

How this eval is graded

Grade against expected.ideal_behavior and expected.rubric. Per-criterion pass requires mean >= 4.0 and no criterion below 3.

Rubric criteria

  • Langchain
  • Ai Platform
  • Agents Langgraph

Recommended for

LangChain (+ LangGraph)LangChain customers

Works with

Related evals

Frequently asked questions

What does the Agents Langgraph eval for LangChain LangChain (+ LangGraph) test?+

Evaluates LangChain's Agents (LangGraph) across 9 scenario-based test cases, each graded against an expected-behavior rubric by an LLM judge, from Corsac's LLM Orchestration Framework eval coverage.

How is the Agents Langgraph eval scored?+

The judge rubric: Grade against expected.ideal_behavior and expected.rubric. Per-criterion pass requires mean >= 4.0 and no criterion below 3.

How many test cases does this eval pack include?+

The Agents Langgraph pack for LangChain LangChain (+ LangGraph) contains 9 test cases. 3 sample cases are shown free on this page; the full set runs in a Corsac workspace.

How do I run this eval?+

Sign up for Corsac, connect your model or agent endpoint, and run the Agents Langgraph pack as-is or after customizing thresholds. Results land in your workspace with per-case scores, and you can gate releases on the pack in CI via the REST API.

Run this eval in your workspace

Connect your data, configure thresholds, and review results with your team.