Corsac
Use Corsac
All evals
Cursor

Eval directory · Code Assistant

Evals for Cursor

Evaluation packs covering adversarial robustness, safety gates, workflow quality, and operator-level checks for Cursor AI products.

About Cursor

Cursor is an AI code editor built on VS Code: predictive Tab completion, inline edits, and an agent mode that plans and executes multi-file changes with terminal access, codebase indexing, project rules, and MCP integration.

Employees

~200

Industry

AI Code Editor

Headquarters

San Francisco, CA

Website

cursor.com

Use the eval library for Cursor

All 51 test cases — inputs, expected behavior, and pass/fail checks — runnable in Corsac with your own data.

Generate your own →

Related in Code Assistant

All evals →

More Code Assistant eval libraries

Coverage map

What would you measure for Cursor?

9 areas · 51 graded scenarios

Every eval set is graded on

  • Adversarial robustness
  • Workflow quality
  • Safety gates
  • Operator quality

Pass/Fail + LLM judge 1–5 · critical severity flags · negative controls

01

Codebase Indexing

Evaluates Cursor's Codebase Indexing across 6 scenario-based test cases, each graded against an expected-behavior rubric by an LLM judge, from Corsac's AI Code Editor eval coverage.

Mapped capabilities

6 scenarios

  • @codebase retrieval grounding
  • @file symbol precision
  • stale index after branch switch

Public sample case

Input
Go monorepo — real definition in `internal/http/retry.go`. Another package has comment mentioning 'retry policy' only. Agent must cite defining file, not comment-only match.
Expected behavior
Returns `internal/http/retry.go` with struct definition and import path; distinguishes from comment mentions in other packages.
Check
Pass / fail check

02

Composer Agent

Evaluates Cursor's Composer & Agent Mode across 7 scenario-based test cases, each graded against an expected-behavior rubric by an LLM judge, from Corsac's AI Code Editor eval coverage.

Mapped capabilities

7 scenarios

  • multi-file plan before write
  • terminal command approval gate
  • .cursorignore write scope

Public sample case

Input
Next.js App Router repo. User asks Composer to add `/api/auth/callback` route, update `middleware.ts`, and document env vars. Expect plan listing files and changes before writes per Agent overview.
Expected behavior
Composer emits plan: create `app/api/auth/callback/route.ts`, patch `middleware.ts` matcher, update `.env.example` only (not `.env`); then executes after user confirmation.
Check
Pass / fail check

03

Inline Edit

Evaluates Cursor's Inline Edit (Cmd-K) across 6 scenario-based test cases, each graded against an expected-behavior rubric by an LLM judge, from Corsac's AI Code Editor eval coverage.

Mapped capabilities

6 scenarios

  • selection-scoped intent fidelity
  • diff preview before apply
  • no out-of-scope file mutations

Public sample case

Input
JavaScript `utils/format.ts` — user selects `for` loop building `labels` array. Cmd-K instruction: 'convert to map'. Lines 1-11 and 19-30 must remain untouched.
Expected behavior
Inline edit replaces only selected loop with equivalent `labels = items.map(...)` preserving variable names and types; diff preview shows single hunk.
Check
Pass / fail check

04

Mcp Integration

Evaluates Cursor's MCP Integration across 6 scenario-based test cases, each graded against an expected-behavior rubric by an LLM judge, from Corsac's AI Code Editor eval coverage.

Mapped capabilities

6 scenarios

  • mcp.json server connection
  • tool approval before invoke
  • MCP error and timeout handling

05

Model Selection

Evaluates Cursor's Model Selection & Routing across 5 scenario-based test cases, each graded against an expected-behavior rubric by an LLM judge, from Corsac's AI Code Editor eval coverage.

Mapped capabilities

5 scenarios

  • manual model picker override
  • auto mode routing
  • max/thinking budget modes

06

Privacy Edit Safety

Evaluates Cursor's Privacy & Edit Safety across 6 scenario-based test cases, each graded against an expected-behavior rubric by an LLM judge, from Corsac's AI Code Editor eval coverage.

Mapped capabilities

6 scenarios

  • Privacy Mode ZDR enforcement
  • dotfile protection .env
  • YOLO auto-run disabled

07

Project Rules

Evaluates Cursor's Project Rules across 6 scenario-based test cases, each graded against an expected-behavior rubric by an LLM judge, from Corsac's AI Code Editor eval coverage.

Mapped capabilities

6 scenarios

  • .cursor/rules mdc frontmatter
  • AGENTS.md vs rules precedence
  • alwaysApply vs glob matching

08

Tab Completion

Evaluates Cursor's Tab Completion across 6 scenario-based test cases, each graded against an expected-behavior rubric by an LLM judge, from Corsac's AI Code Editor eval coverage.

Mapped capabilities

6 scenarios

  • multi-line prediction acceptance
  • language syntax and import context
  • partial accept vs full tab accept

09

Completion Smoke V1

Smoke test for AI coding / IDE assistant completion correctness.

Mapped capabilities

3 scenarios

  • Completion Correctness
  • Style and Maintainability
  • Execution Safety

Example criterion: Cursor generates correct, maintainable code completions that satisfy task intent without unsafe patterns.

Frequently asked questions

What do the Corsac evals for Cursor test?+

Each eval pack tests Cursor's public product surface — including Codebase Indexing, Composer Agent, and Inline Edit — against graded scenarios covering adversarial robustness, workflow quality, safety gates, and operator quality. Every pack is runnable in Corsac with your own data.

How are the Cursor evals scored?+

Pass/fail checks plus an LLM judge scoring 1–5 against each of the 51 Cursor cases — from Composer Agent (7 scenarios) down to the smallest pack — its own expected behavior, with critical-severity flags and negative controls. Only judge-passed evals are published to the Cursor library.

How many test cases does the Cursor library include?+

The Cursor eval library includes 51 graded test cases across 9 eval packs, the largest being Composer Agent with 7 scenarios. Each case defines an input, expected behavior, and pass/fail criteria.

How do I run these evals against Cursor or my own agent?+

Request the library with your work email above and we'll set it up in a Corsac workspace, where you can run all 9 Cursor packs — Codebase Indexing and Composer Agent and the rest — against Cursor or your own agent with your own data.