Memories And Rules
Windsurf · Windsurf
AI Code Editor — Windsurf (Codeium)
Evaluates Windsurf's Memories & Rules across 9 scenario-based test cases, each graded against an expected-behavior rubric by an LLM judge, from Corsac's AI Code Editor eval coverage.
About Windsurf
Windsurf (by Codeium) is an agentic AI IDE. Its Cascade agent does multi-file, plan-and-act coding with terminal access, alongside predictive Tab / Supercomplete completions, local codebase indexing and @-mentions, persistent Memories and .windsurfrules, Flows that keep the AI and human in shared state, MCP integrations, and a multi-model picker.
Sample tests· showing 3 of 9
| # | Input | Expected behavior | Check |
|---|---|---|---|
| 01 | In an earlier session the user pinned a Memory: 'this project uses pnpm, never npm'. In a new session Cascade needs to install a dependency. | Recall the pinned Memory and use pnpm for the install in the new session without being re-told. Persistent Memories should survive across sessions and steer behavior; Cascade should not default back to npm in violation of the stored Memory. | Pass / FailCode Assistanthigh |
| 02 | Cascade auto-generates a Memory from the conversation. The user corrected a fact mid-conversation ('actually we use Postgres, not MySQL'). | The auto-generated Memory should capture the corrected fact (Postgres), not the superseded one (MySQL). Auto-memories must reflect the latest accurate state of the conversation and not encode a fact the user explicitly retracted. | Pass / FailCode Assistanthigh |
| 03 | The workspace .windsurfrules states 'all new React components must be function components with TypeScript; no class components'. The user asks for a new component. | Generate a TypeScript function component per the rule. Cascade must honor .windsurfrules constraints in its output; producing a class component or a plain-JS component violates the stated rule. | Pass / FailCode Assistanthigh |
How this eval is graded
Grade against expected.ideal_behavior and expected.rubric. Per-criterion pass requires mean >= 4.0 and no criterion below 3.
Rubric criteria
- Windsurf
- Code Assistant
- Memories And Rules
Recommended for
Works with
Related evals
Browserbase
Evaluates Browserbase's Captcha Handling across scenario-based test cases, each graded against an expected-behavior rubric by an LLM judge, from Corsac's Browser infrastructure eval coverage.
View Code AssistantBrowserbase
Evaluates Browserbase's Concurrency & Rate Limits across scenario-based test cases, each graded against an expected-behavior rubric by an LLM judge, from Corsac's Browser infrastructure eval coverage.
View Code AssistantBrowserbase
Evaluates Browserbase's Live Debugging & Session Inspector across scenario-based test cases, each graded against an expected-behavior rubric by an LLM judge, from Corsac's Browser infrastructure eval coverage.
ViewFrequently asked questions
What does the Memories And Rules eval for Windsurf Windsurf test?+
Evaluates Windsurf's Memories & Rules across 9 scenario-based test cases, each graded against an expected-behavior rubric by an LLM judge, from Corsac's AI Code Editor eval coverage.
How is the Memories And Rules eval scored?+
The judge rubric: Grade against expected.ideal_behavior and expected.rubric. Per-criterion pass requires mean >= 4.0 and no criterion below 3.
How many test cases does this eval pack include?+
The Memories And Rules pack for Windsurf Windsurf contains 9 test cases. 3 sample cases are shown free on this page; the full set runs in a Corsac workspace.
How do I run this eval?+
Sign up for Corsac, connect your model or agent endpoint, and run the Memories And Rules pack as-is or after customizing thresholds. Results land in your workspace with per-case scores, and you can gate releases on the pack in CI via the REST API.
Run this eval in your workspace
Connect your data, configure thresholds, and review results with your team.