Composio
For ComposioAI Platform

Framework Integrations

Composio · Composio

Agent Tooling & Integrations — Composio

Evaluates Composio's Framework Integrations across 9 scenario-based test cases, each graded against an expected-behavior rubric by an LLM judge, from Corsac's Agent Tooling & Integrations eval coverage.

About Composio

Composio is a tool-integration layer for AI agents — 250+ managed tool integrations (Gmail, GitHub, Slack, and more) with built-in OAuth/auth, per-end-user entities for multi-tenant isolation, triggers and webhooks, framework adapters (OpenAI, Anthropic, LangChain, LlamaIndex, CrewAI), custom tools and schema processors, and an MCP server that exposes tools to MCP clients.

Employees

~40

Industry

Agent Tooling

Headquarters

San Francisco, CA

Sample tests· showing 3 of 9

#InputExpected behaviorCheck
01

Using the OpenAI adapter, the model returns a tool_calls array. The integrator must run the Composio actions and continue the conversation.

Hand each tool_call to Composio's handler to execute the corresponding action, then append one tool-role message per tool_call_id with the result and call the model again. Match results to tool_call_id exactly; loop until the model returns a final message.

Pass / FailAi Platformhigh
02

A Composio action with nested object arguments is exposed to OpenAI and to LangChain; the LangChain path silently flattens the nested args.

Ensure the adapter preserves the action's full input schema (nested objects, required fields, enums) in each framework's tool format so the model fills arguments correctly. Validate that the rendered schema round-trips back to a valid execute payload.

Pass / FailAi Platformmedium
03

Using the Anthropic adapter, the assistant turn contains tool_use blocks for Composio actions.

Execute each tool_use via Composio and return a user turn with one tool_result block per tool_use_id, preserving role alternation. Keep tool_use/tool_result pairing intact and resend prior assistant blocks unmodified on the next call.

Pass / FailAi Platformhigh

Unlock full benchmark

6 more test cases

Use this benchmark

How this eval is graded

Grade against expected.ideal_behavior and expected.rubric. Per-criterion pass requires mean >= 4.0 and no criterion below 3.

Rubric criteria

  • Composio
  • Ai Platform
  • Framework Integrations

Recommended for

ComposioComposio customers

Works with

Related evals

Frequently asked questions

What does the Framework Integrations eval for Composio Composio test?+

Evaluates Composio's Framework Integrations across 9 scenario-based test cases, each graded against an expected-behavior rubric by an LLM judge, from Corsac's Agent Tooling & Integrations eval coverage.

How is the Framework Integrations eval scored?+

The judge rubric: Grade against expected.ideal_behavior and expected.rubric. Per-criterion pass requires mean >= 4.0 and no criterion below 3.

How many test cases does this eval pack include?+

The Framework Integrations pack for Composio Composio contains 9 test cases. 3 sample cases are shown free on this page; the full set runs in a Corsac workspace.

How do I run this eval?+

Sign up for Corsac, connect your model or agent endpoint, and run the Framework Integrations pack as-is or after customizing thresholds. Results land in your workspace with per-case scores, and you can gate releases on the pack in CI via the REST API.

Run this eval in your workspace

Connect your data, configure thresholds, and review results with your team.