Eval directory

Evals for Supermemory

Eval coverage for Supermemory, mapped from its public product surface.

About Supermemory

Supermemory is a hosted "context cloud" that gives AI agents persistent memory, retrieval, user profiles, connectors, and extractors behind one API. It packages primitives including a knowledge-graph memory layer, hybrid-search RAG, a POSIX filesystem mount (SMFS), and OAuth connectors to tools like Slack, Notion, Drive, and Gmail. It ships TypeScript and Python SDKs, an MCP server for clients such as Claude Code and Cursor, usage-based paid tiers, and a self-hosting option.

Industry

memory and context infrastructure (API) for AI agents

Website

supermemory.ai

Use the eval library for Supermemory

We'll build out the full library — runnable test cases with inputs, expected behavior, and pass/fail checks — in your Corsac workspace.

All evals →

More AI eval libraries

Coverage map

What would you measure for Supermemory?

6 scoring areas · 24 capabilities mapped · grounded in 8 cited pages

Every eval set is graded on

Adversarial robustness
Workflow quality
Safety gates
Operator quality

Pass/Fail + LLM judge 1–5 · critical severity flags · negative controls

Memory & Continual Learning

The knowledge-graph memory layer: how memory entries are created, updated, forgotten, and connected over time, including dynamic dreaming as a default behavior.

Mapped capabilities

4 capabilities

Memory entry lifecycle
Create, list, update, and forget extracted memory entries via the Memories endpoints.
Knowledge-graph structure
Documents as nodes, memories as linked entries, and the relationships surfaced between them.
Dynamic dreaming behavior
Automatic connection-forming across stored context and where it applies (API, plugins, agents).
Memory vs. working context
When an agent should reach into stored memory rather than rely on what is already in context.

SuperRAG Retrieval

The hybrid-search retrieval layer that returns structured context for documents and memories under stated low-latency targets.

“Hybrid search, reranking, and structured context for documents at sub-300ms latency.” supermemory.ai

Mapped capabilities

4 capabilities

Search mode selection
Choosing memories, documents, or hybrid recall on POST /v4/search.
Hybrid search and reranking
Combining semantic and keyword signals plus reranking into ordered results.
Structured context assembly
Shaping retrieved chunks into context an agent can consume.
Latency expectations
Sub-300ms retrieval framing and what it does and does not cover.

Ingestion & Document Lifecycle

Getting content into the pipeline and knowing when it is actually queryable — documents, files, batches, and conversations.

Mapped capabilities

4 capabilities

Ingest entrypoints
POST /v3/documents and the SDK client.add path for documents, files, batches, and conversations.
Processing status
Polling GET /v3/documents/{id} until status is done before searching.
Document operations
List, update, delete, chunk inspection, and file URL retrieval.
Multi-format indexing
Automatic indexing of arbitrary file formats as described for the filesystem and ingest paths.

Illustrative example

Input: I just called client.add with a PDF and my next /v4/search call comes back empty. Is search broken, or am I missing a step?
Expected behavior: Explains that ingestion is asynchronous, so the document is not immediately searchable. Directs the developer to poll GET /v3/documents/{id} until status is "done", then retry POST /v4/search, rather than treating the empty result as a search defect.

Profiles & Container Scoping

Per-user and per-entity context that stays coherent across sessions, and the container tag that scopes ingest, search, and profiles to one tenant.

“Supermemory gives your agents state-of-the-art memory, RAG, user profiles, connectors, and extractors, all built in.” supermemory.ai

Mapped capabilities

4 capabilities

Profile retrieval
POST /v4/profile returning static and dynamic facts for a container.
containerTag consistency
Using the same tag across ingest, search, and profiles so one engine serves multiple outputs.
Multi-tenant settings
Container tag settings, merge, and delete operations.
Cross-session identity
Preference, behavior, and identity context persisting beyond a single conversation.

Connectors & Filesystems

Bringing external sources in through OAuth connectors, and exposing memory as a real POSIX mount agents can navigate with standard tools.

“Native POSIX on macOS and Linux; your agent just uses ls, cat, and grep.” supermemory.ai

Mapped capabilities

4 capabilities

OAuth connections and sync
Connecting and syncing Slack, Notion, Drive, Gmail, GitHub, S3, and web crawler sources.
Tier-gated connector access
Which connectors are available on Free, Pro, Max, and Scale.
SMFS POSIX mount
Native macOS and Linux mount where agents use ls, cat, and grep.
Semantic grep and profile.md
grep resolving to semantic search and a live profile.md synthesized on the fly.

Developer Surfaces & Deployment

How teams actually integrate and operate Supermemory: SDKs, the MCP server, the memory-graph component, auth, and hosted versus self-hosted deployment.

Mapped capabilities

4 capabilities

SDK usage
TypeScript (npm install supermemory) and Python (pip install supermemory) clients over the same API.
MCP server setup
npx mcp-remote against mcp.supermemory.ai, OAuth authentication, and multi-client access.
Auth and base URL
Bearer sm_ API keys and switching between api.supermemory.ai and a self-hosted instance URL.
Memory Graph component
@supermemory/memory-graph React visualization backed by a documents API route.

Illustrative example

Input: We're running Supermemory ourselves at http://localhost:6767. Should the SDK still call api.supermemory.ai, and do I keep the Bearer API key?
Expected behavior: States that the base URL must be overridden to the self-hosted instance URL instead of https://api.supermemory.ai, while endpoints still authenticate with an Authorization: Bearer key. Points to the self-hosting docs rather than inventing configuration details.

Coverage is mapped from Supermemory's public pages (8 crawled). Examples are illustrative, not real test cases. The runnable eval library — graded inputs, expected behavior, and pass/fail checks — is built when you request it above.

Frequently asked questions

What do the Corsac evals for Supermemory test?+

The coverage map is generated from Supermemory's own public product surface (memory and context infrastructure (API) for AI agents): 6 scoring areas — Memory & Continual Learning, SuperRAG Retrieval, and Ingestion & Document Lifecycle, and more — spanning 24 mapped capabilities, each graded on adversarial robustness, workflow quality, safety gates, and operator quality once the library is built.

How are the Supermemory evals scored?+

Every case generated for Supermemory — across Memory & Continual Learning and SuperRAG Retrieval and the other mapped areas — is graded with pass/fail checks plus an LLM judge scoring 1–5 against its expected behavior, with critical-severity flags and negative controls. Only judge-passed evals are published.

How many test cases does the Supermemory library include?+

The full Supermemory library is built on request. The coverage map spans 6 areas and 24 capabilities (for example, Memory entry lifecycle and Knowledge-graph structure under Memory & Continual Learning); each becomes graded test cases — inputs, expected behavior, pass/fail checks — in your Corsac workspace.

How do I run these evals against Supermemory or my own agent?+

Request the library with your work email above. We'll build out all 6 mapped Supermemory areas and set them up in a Corsac workspace, where you can run every test case against Supermemory or your own agent with your own data.