Tool Use And Function Orchestration
Devin · Cognition
Autonomous Coding Agent — Cognition AI (Devin)
Evaluates Cognition's Tool Use & Function Orchestration across 9 scenario-based test cases, each graded against an expected-behavior rubric by an LLM judge, from Corsac's Autonomous Coding Agent eval coverage.
About Cognition
Cognition builds Devin, an autonomous AI software engineer that plans, writes, debugs, and ships code in a sandboxed cloud environment with terminal, browser, and editor access, session continuity, and human-in-the-loop review.
Sample tests· showing 3 of 9
| # | Input | Expected behavior | Check |
|---|---|---|---|
| 01 | Devin needs to run a Bash-specific command (`source ./.env`) but the sandbox's default shell is sh. | Explicitly invoke `bash -lc '...'` for Bash-specific syntax or normalize scripts to POSIX sh. Do not assume the default shell is Bash; capture and check the actual shell at session start. On `source: not found` errors, fall back to the explicit invocation rather than retrying. | Pass / FailCode Assistantmedium |
| 02 | Operator connects an MCP server (e.g., Linear MCP) to the session so Devin can fetch ticket context. The MCP server returns a malformed response. | Surface the MCP error to Devin's reasoning as a tool error with the upstream message verbatim. Do not swallow into a generic 'tool failed' string. Cap retries (e.g., 2) before falling back to asking the operator for the missing context. Log MCP errors with the server id for diagnosis. | Pass / FailCode Assistanthigh |
| 03 | Devin's browser action fails: page navigation times out after 10s on a slow single-page app. | Increase wait-for-load condition to a documented signal (e.g., specific DOM element present) rather than retrying the raw navigation. Retry at most 2x; after that, escalate to operator as a possible app-side outage with the URL and last-screenshot evidence. | Pass / FailCode Assistantmedium |
How this eval is graded
Grade against expected.ideal_behavior and expected.rubric. Per-criterion pass requires mean >= 4.0 and no criterion below 3.
Rubric criteria
- Cognition
- Code Assistant
- Tool Use And Function Orchestration
Recommended for
Works with
Related evals
Browserbase
Evaluates Browserbase's Captcha Handling across scenario-based test cases, each graded against an expected-behavior rubric by an LLM judge, from Corsac's Browser infrastructure eval coverage.
View Code AssistantBrowserbase
Evaluates Browserbase's Concurrency & Rate Limits across scenario-based test cases, each graded against an expected-behavior rubric by an LLM judge, from Corsac's Browser infrastructure eval coverage.
View Code AssistantBrowserbase
Evaluates Browserbase's Live Debugging & Session Inspector across scenario-based test cases, each graded against an expected-behavior rubric by an LLM judge, from Corsac's Browser infrastructure eval coverage.
ViewFrequently asked questions
What does the Tool Use And Function Orchestration eval for Cognition Devin test?+
Evaluates Cognition's Tool Use & Function Orchestration across 9 scenario-based test cases, each graded against an expected-behavior rubric by an LLM judge, from Corsac's Autonomous Coding Agent eval coverage.
How is the Tool Use And Function Orchestration eval scored?+
The judge rubric: Grade against expected.ideal_behavior and expected.rubric. Per-criterion pass requires mean >= 4.0 and no criterion below 3.
How many test cases does this eval pack include?+
The Tool Use And Function Orchestration pack for Cognition Devin contains 9 test cases. 3 sample cases are shown free on this page; the full set runs in a Corsac workspace.
How do I run this eval?+
Sign up for Corsac, connect your model or agent endpoint, and run the Tool Use And Function Orchestration pack as-is or after customizing thresholds. Results land in your workspace with per-case scores, and you can gate releases on the pack in CI via the REST API.
Run this eval in your workspace
Connect your data, configure thresholds, and review results with your team.