Cognition
For CognitionCode Assistant

Devin Sessions And Planning

Devin · Cognition

Autonomous Coding Agent — Cognition AI (Devin)

Evaluates Cognition's Devin Sessions & Planning across 9 scenario-based test cases, each graded against an expected-behavior rubric by an LLM judge, from Corsac's Autonomous Coding Agent eval coverage.

About Cognition

Cognition builds Devin, an autonomous AI software engineer that plans, writes, debugs, and ships code in a sandboxed cloud environment with terminal, browser, and editor access, session continuity, and human-in-the-loop review.

Employees

~200

Industry

Autonomous Coding Agent

Headquarters

San Francisco, CA

Sample tests· showing 3 of 9

#InputExpected behaviorCheck
01

Operator creates a Devin session via POST /v1/sessions passing snapshot_id from a prior session to resume on the same VM state.

Pass snapshot_id only when the prior session was NOT terminated — terminate is irreversible and invalidates the snapshot. On 4xx referencing an invalid snapshot, surface to operator with a 'create fresh session' fallback, do not loop the retry. Record the new session_id and persist the snapshot lin…

Pass / FailCode Assistanthigh
02

Operator inspects a long-running Devin session and expects to see Devin's plan BEFORE execution begins so they can intervene on misinterpretation.

Surface the structured plan (milestone list with dependencies) in the session UI / API before file mutations or terminal commands run. Pause-at-plan-confirm is opt-in per playbook; default is execute-after-plan with the plan visible. Do not collapse plan and execute into one opaque step.

Pass / FailCode Assistanthigh
03

Operator sends a follow-up instruction to an active session via POST /v1/sessions/{id}/message ('also add a unit test for the new branch').

Follow-up message appends to the same session — Devin folds the new instruction into the existing plan and amends milestones rather than restarting. Confirm session_id is still in a non-terminal state before sending. On 404 / terminated, prompt operator to create a new session with snapshot_id.

Pass / FailCode Assistanthigh

Unlock full benchmark

6 more test cases

Use this benchmark

How this eval is graded

Grade against expected.ideal_behavior and expected.rubric. Per-criterion pass requires mean >= 4.0 and no criterion below 3.

Rubric criteria

  • Cognition
  • Code Assistant
  • Devin Sessions And Planning

Recommended for

DevinCognition customers

Works with

Related evals

Frequently asked questions

What does the Devin Sessions And Planning eval for Cognition Devin test?+

Evaluates Cognition's Devin Sessions & Planning across 9 scenario-based test cases, each graded against an expected-behavior rubric by an LLM judge, from Corsac's Autonomous Coding Agent eval coverage.

How is the Devin Sessions And Planning eval scored?+

The judge rubric: Grade against expected.ideal_behavior and expected.rubric. Per-criterion pass requires mean >= 4.0 and no criterion below 3.

How many test cases does this eval pack include?+

The Devin Sessions And Planning pack for Cognition Devin contains 9 test cases. 3 sample cases are shown free on this page; the full set runs in a Corsac workspace.

How do I run this eval?+

Sign up for Corsac, connect your model or agent endpoint, and run the Devin Sessions And Planning pack as-is or after customizing thresholds. Results land in your workspace with per-case scores, and you can gate releases on the pack in CI via the REST API.

Run this eval in your workspace

Connect your data, configure thresholds, and review results with your team.