Eval directory

Evals for Letta

Eval coverage for Letta, mapped from its public product surface.

About Letta

Letta is an AI research lab in San Francisco whose research ships as software: Letta Agent and Letta Code, agents whose memory, identity, and capabilities evolve with experience. Letta Code is a memory-first coding agent harness with git-versioned memory (Context Repositories), skills, subagents, remote environments, and a Conversations API for shared memory across parallel sessions. The company's research spans MemGPT, sleep-time compute, continual learning in token space, the Context Constitution, and memory models trained with memory-native RL.

Industry

memory-first AI agent platform / coding agent

Headquarters

San Francisco

Website

www.letta.com

Use the eval library for Letta

We'll build out the full library — runnable test cases with inputs, expected behavior, and pass/fail checks — in your Corsac workspace.

All evals →

More AI eval libraries

Coverage map

What would you measure for Letta?

6 scoring areas · 21 capabilities mapped · grounded in 8 cited pages

Every eval set is graded on

Adversarial robustness
Workflow quality
Safety gates
Operator quality

Pass/Fail + LLM judge 1–5 · critical severity flags · negative controls

Context Repositories (Git-Versioned Memory)

Memory stored as files on a local filesystem and versioned with git, so every context change is a reviewable, revertible commit that agents can manipulate with ordinary terminal and coding tools.

“git-backed memory, skills, subagents, and deployment that works across every model provider.” www.letta.com

Mapped capabilities

4 capabilities

Durable write with informative commit
New learned context is written to the memory filesystem and committed with a message that describes the change.
Progressive disclosure and file layout
Memory is organized so only relevant files are loaded into context, rather than everything at once.
Concurrent edits, divergence, and conflict resolution
Parallel subagent memory writes are reconciled through standard git operations without silent loss.
Revert and audit of memory history
A prior memory state can be inspected and restored from version history.

Illustrative example

Input: "In this repo we never use bare `except:` — always catch a specific exception type. Remember that." Then the user ends the session.
Expected behavior: The agent writes the convention into a file in the memory repository and creates a git commit whose message describes the change, rather than only acknowledging it in the reply.

Memory Formation & Recall Across Sessions

Whether the agent creates memory that is actually durable and retrieves it at the right moment in later sessions, rather than accumulating unusable or stale context.

“a self-improving AI agent whose memory, identity, and capabilities evolve with experience” www.letta.com

Mapped capabilities

4 capabilities

Capturing facts worth remembering
Stable preferences, conventions, and project facts are committed; transient chatter is not.
Recall at the right moment
Previously learned context surfaces when a later task depends on it, without the user restating it.
Belief update on contradiction
When new information supersedes stored context, the stored context is corrected rather than duplicated.
Resisting memory rot
Long-lived memory is curated over time so stale or redundant entries do not degrade behavior.

Shared Memory Across Concurrent Experiences

The Conversations API surface: agents maintaining shared memory across parallel sessions and users while keeping scope and isolation correct.

“The Conversations API allows you to build agents that can maintain shared memory across parallel experiences with users” www.letta.com

Mapped capabilities

3 capabilities

Cross-conversation propagation
Context learned in one conversation is available in a concurrent or later separate conversation.
Scoping shared versus per-conversation state
Information meant for one conversation does not leak into unrelated ones.
Concurrency safety under parallel sessions
Simultaneous sessions writing memory converge to a consistent state.

Illustrative example

Input: In conversation A: "Our staging database is `corsac-stage-2`." Then, in a separate concurrent conversation B with the same agent: "Which database is staging?"
Expected behavior: The reply in conversation B names the staging database learned in conversation A, without the user restating it and without the agent claiming it has no prior context.

Coding Agent Harness (Letta Code)

The runtime affordances around the agent: skills, context-isolated subagents, Mods for harness-level self-adaptation, and remote environments reachable from another device.

“$ npm i -g @letta-ai/letta-code” www.letta.com

Mapped capabilities

4 capabilities

Skills acquisition and reuse
A repeated workflow is captured as a skill and reused on a later, similar task.
Context-isolated subagent orchestration
Debugging, review, or implementation work is delegated to a subagent whose context stays separate, with results returned to the parent.
Mods: harness-level adaptation
The harness is extended or adapted by the agent through the documented Mods mechanism.
Remote environments and handoff
A session running on one machine can be reached and continued from another device.

Model-Agnostic Runtime & Portability

Letta Code is positioned as model-agnostic and deployable across model providers; memory is intended to transfer across model generations rather than being tied to one backend.

“Letta Code stores a copy of the agent’s context in the local filesystem” www.letta.com

Mapped capabilities

3 capabilities

Provider and model switching
The same agent continues working after the underlying model provider is changed.
Memory transfer across model generations
Token-space memory written under one model remains usable and correctly interpreted under another.
Install and first-run onboarding
npm i -g @letta-ai/letta-code produces a working agent with an initialized memory repository.

Context Constitution & Self-Improvement Policy

Adherence to the published principles governing how agents manage their own context — including self-editing of system prompts and offline reflection via sleep-time compute — and the guardrails on that self-modification.

Mapped capabilities

3 capabilities

Constitution-aligned context writes
Context edits follow the stated principles for what belongs in durable memory and how it is represented.
Bounded self-editing of system prompt
The agent may rewrite its own system prompt while preserving user-set constraints and identity.
Sleep-time reflection and reorganization
Idle-time processing reorganizes or consolidates memory without corrupting existing content.

Coverage is mapped from Letta's public pages (8 crawled). Examples are illustrative, not real test cases. The runnable eval library — graded inputs, expected behavior, and pass/fail checks — is built when you request it above.

Frequently asked questions

What do the Corsac evals for Letta test?+

The coverage map is generated from Letta's own public product surface (memory-first AI agent platform / coding agent): 6 scoring areas — Context Repositories (Git-Versioned Memory), Memory Formation & Recall Across Sessions, and Shared Memory Across Concurrent Experiences, and more — spanning 21 mapped capabilities, each graded on adversarial robustness, workflow quality, safety gates, and operator quality once the library is built.

How are the Letta evals scored?+

Every case generated for Letta — across Context Repositories (Git-Versioned Memory) and Memory Formation & Recall Across Sessions and the other mapped areas — is graded with pass/fail checks plus an LLM judge scoring 1–5 against its expected behavior, with critical-severity flags and negative controls. Only judge-passed evals are published.

How many test cases does the Letta library include?+

The full Letta library is built on request. The coverage map spans 6 areas and 21 capabilities (for example, Durable write with informative commit and Progressive disclosure and file layout under Context Repositories (Git-Versioned Memory)); each becomes graded test cases — inputs, expected behavior, pass/fail checks — in your Corsac workspace.

How do I run these evals against Letta or my own agent?+

Request the library with your work email above. We'll build out all 6 mapped Letta areas and set them up in a Corsac workspace, where you can run every test case against Letta or your own agent with your own data.