Eval directory

Evals for Redis

Eval coverage for Redis, mapped from its public product surface.

About Redis

Redis is an in-memory data platform for real-time applications and AI agents, offering vector search, semantic caching, agent memory, and JSON/streams data structures. It is available as open source, as fully managed Redis Cloud on AWS, Azure, and GCP, and as self-managed Redis Software for on-prem, hybrid, and Kubernetes deployments. AI-focused capabilities include the Redis Iris context engine, Context Retriever for retrieval over structured and unstructured data, LangCache semantic caching, and Redis Data Integration for near-real-time sync from source databases.

Industry

real-time data platform and vector database for AI agents and apps

Website

redis.io

Use the eval library for Redis

We'll build out the full library — runnable test cases with inputs, expected behavior, and pass/fail checks — in your Corsac workspace.

All evals →

More AI eval libraries

Coverage map

What would you measure for Redis?

6 scoring areas · 24 capabilities mapped · grounded in 8 cited pages

Every eval set is graded on

Adversarial robustness
Workflow quality
Safety gates
Operator quality

Pass/Fail + LLM judge 1–5 · critical severity flags · negative controls

Vector search and semantic retrieval

Storing and querying vector embeddings with associated metadata for semantic search over unstructured data, via Redis Search and the RedisVL client library.

“Redis Data Integration keeps Redis in sync with the primary database in near real time.” redis.io

Mapped capabilities

4 capabilities

Vector indexing over hashes and JSON documents
Choosing hash vs. JSON storage for embeddings and defining an index over vector plus metadata fields.
KNN and range queries
Formulating nearest-neighbor and range vector queries and interpreting returned scores/results.
Metadata filtering on vector queries
Combining structured metadata predicates with vector similarity in a single query.
RedisVL client usage
Using the dedicated Python vector library to define schemas, load vectors, and run searches.

Illustrative example

Input: Our embeddings live in JSON documents alongside a category field. How do we run a nearest-neighbor search that only returns documents where category is "support"?
Expected behavior: Explains that a vector index can be defined over JSON documents with both the vector field and the category metadata field, then a KNN query is issued with a filter on category so similarity ranking is restricted to matching documents.

AI context engine and agent memory

Redis capabilities aimed at AI agents: the Iris context engine, Context Retriever over structured and unstructured data, LangCache semantic caching, and short- and long-term agent memory.

Mapped capabilities

4 capabilities

Redis Iris as a real-time context engine
What Iris covers (memory, retrieval, state, streaming) and when it applies to an AI app.
Context Retriever over structured and unstructured data
Turning business data into governed tools agents can query for relevant context.
LangCache semantic caching
Reusing cached responses for semantically similar prompts to reduce LLM API cost.
Short-term session vs. long-term persistent memory
Distinguishing per-session agent memory from memory that survives across interactions.

Core data structures and developer APIs

The foundational Redis surface used by application developers: caching and primary-database usage, JSON documents, streams and pub/sub, and command compatibility across language clients.

“Redis Cloud is a fully managed database-as-a-service” redis.io

Mapped capabilities

4 capabilities

Cache, primary database, and custom database setup
Picking the right starting configuration for a new Redis workload.
JSON document storage and querying
Storing, indexing, and querying JSON documents.
Streams and pub/sub event processing
Real-time data streaming and event-driven patterns.
Language client selection and idioms
redis-py, node-redis, Jedis, Lettuce, StackExchange.Redis, Go, Rust, PHP, C and their command compatibility.

Deployment models and plan fit

Choosing among Redis Open Source, Redis Cloud, Redis Software (on-prem/hybrid), and Redis for Kubernetes, and matching a workload to the right commercial tier.

“Tiered RAM + flash storage for TB-scale at up to 80% less cost.” redis.io

Mapped capabilities

4 capabilities

Open source vs. Cloud vs. Software vs. Kubernetes
Matching deployment model to environment, support needs, and operational ownership.
Free, Essentials, and Pro tier boundaries
Capacity, single vs. multiple DBs, SLA levels, and features gated by tier.
Active-Active multi-region replication
Geo-distributed writes, local-speed reads, and failover expectations.
Redis Flex tiered RAM plus flash
Using tiered storage for TB-scale datasets and adjusting the RAM:flash ratio.

Illustrative example

Input: We need Active-Active multi-region writes on AWS and are targeting 99.999% uptime. Which Redis Cloud plan should we choose?
Expected behavior: Recommends the Pro (dedicated) plan, since Active-Active multi-region and up to 99.999% uptime are Pro capabilities, and notes that Essentials is a shared, single-database tier topping out at up to 99.99% uptime.

Redis Cloud operations and security

Day-two management of managed Redis: creating and editing databases, subscriptions, monitoring, account settings, security controls, and the REST API.

“SAML SSO, RBAC, encryption in transit, encryption at rest” redis.io

Mapped capabilities

4 capabilities

Database and subscription lifecycle
Creating, viewing, editing, and managing databases within a subscription.
Performance monitoring and scaling
Monitoring database performance and applying auto-scaling or plan changes.
Access control and encryption
SAML SSO, RBAC, private connectivity, and encryption in transit and at rest.
REST API automation
Programmatic provisioning and management of Cloud resources.

Data integration, migration, and tooling

Getting data into and out of Redis and inspecting it: Redis Data Integration, migration paths, and the client tooling ecosystem.

Mapped capabilities

4 capabilities

Redis Data Integration near-real-time sync
Keeping Redis in sync with a source database or warehouse, with data shaped for fast reads.
Migration into Redis Cloud
Moving an existing deployment onto the managed service.
Redis Insight and CLI workflows
Visualizing, querying, and optimizing data through the GUI, CLI, and VS Code tooling.
Redis MCP and framework integrations
Exposing Redis to MCP clients and connecting through LangChain, Amazon Bedrock, or Spring Data.

Coverage is mapped from Redis's public pages (8 crawled). Examples are illustrative, not real test cases. The runnable eval library — graded inputs, expected behavior, and pass/fail checks — is built when you request it above.

Frequently asked questions

What do the Corsac evals for Redis test?+

The coverage map is generated from Redis's own public product surface (real-time data platform and vector database for AI agents and apps): 6 scoring areas — Vector search and semantic retrieval, AI context engine and agent memory, and Core data structures and developer APIs, and more — spanning 24 mapped capabilities, each graded on adversarial robustness, workflow quality, safety gates, and operator quality once the library is built.

How are the Redis evals scored?+

Every case generated for Redis — across Vector search and semantic retrieval and AI context engine and agent memory and the other mapped areas — is graded with pass/fail checks plus an LLM judge scoring 1–5 against its expected behavior, with critical-severity flags and negative controls. Only judge-passed evals are published.

How many test cases does the Redis library include?+

The full Redis library is built on request. The coverage map spans 6 areas and 24 capabilities (for example, Vector indexing over hashes and JSON documents and KNN and range queries under Vector search and semantic retrieval); each becomes graded test cases — inputs, expected behavior, pass/fail checks — in your Corsac workspace.

How do I run these evals against Redis or my own agent?+

Request the library with your work email above. We'll build out all 6 mapped Redis areas and set them up in a Corsac workspace, where you can run every test case against Redis or your own agent with your own data.