Eval directory
Evals for Exa Labs
7 evaluation packs covering adversarial robustness, safety gates, workflow quality, and operator-level checks for Exa Labs AI products.
About Exa Labs
Exa (formerly Metaphor) is a neural search API that understands the meaning of queries rather than matching keywords — returning the most relevant URLs and content from the web for any semantic question. Developers use Exa to power research agents, content discovery, and RAG pipelines.
How complete this published benchmark library is across datasets, metrics, rubrics, use-case maps, and pack context. This is library coverage, not an agent performance score.
Test datasets
7/7 packs
Scoring metrics
0/7 packs
Judge rubrics
7/7 packs
Use-case maps
0/7 packs
Pack context
7/7 packs
Available eval packs for Exa Labs
7 packs ready to run.
Contents Freshness Cache Semantics
Evaluates Exa Labs' Contents Freshness & Cache Semantics across 9 scenario-based test cases, each graded against an expected-behavior rubric by an LLM judge, from Corsac's Neural web search API eval coverage.
Corpus Safety Prompt Injection
Prompt InjectionEvaluates Exa Labs' Corpus Safety & Prompt Injection across 12 scenario-based test cases, each graded against an expected-behavior rubric by an LLM judge, from Corsac's Neural web search API eval coverage.
Filters Domains Date Controls
Evaluates Exa Labs' Filters Domains & Date Controls across 10 scenario-based test cases, each graded against an expected-behavior rubric by an LLM judge, from Corsac's Neural web search API eval coverage.
Find Similar Semantic Neighbors
Evaluates Exa Labs' Find Similar & Semantic Neighbors across 8 scenario-based test cases, each graded against an expected-behavior rubric by an LLM judge, from Corsac's Neural web search API eval coverage.
Governance Pii Copyright Rate Limits
PII LeakageEvaluates Exa Labs' Governance PII Copyright & Rate Limits across 7 scenario-based test cases, each graded against an expected-behavior rubric by an LLM judge, from Corsac's Neural web search API eval coverage.
Highlights Summary Synthesis Grounding
Answer RelevanceEvaluates Exa Labs' Highlights Summary & Synthesis Grounding across 12 scenario-based test cases, each graded against an expected-behavior rubric by an LLM judge, from Corsac's Neural web search API eval coverage.
Search Mode Selection Query Routing
Evaluates Exa Labs' Search Mode Selection & Query Routing across 10 scenario-based test cases, each graded against an expected-behavior rubric by an LLM judge, from Corsac's Neural web search API eval coverage.
Why eval Exa Labs AI
Exa Labs's AI features ship behind brand promises about accuracy, safety, and reliability. Buyers and integrators need to know those promises hold up under adversarial prompts, edge-case workflows, and the long tail of real customer inputs — not just the demo path.
The Corsac eval library for Exa Labs measures four dimensions teams care about most when deploying search & knowledge agents:
- Adversarial robustness — does the agent resist prompt injection, jailbreaks, and social-engineering attempts?
- Workflow quality— does it complete the task buyers were shown in the demo, on inputs that don't look like the demo?
- Safety gates — does it escalate or refuse when it should, and only then?
- Operator quality — does it preserve analyst trust by surfacing the right context at the right time?
Every eval pack above is hand-authored against Exa Labs's public product surface and runnable in Corsac with your own data.