Eval directory
Evals for Vercel AI SDK
8 evaluation packs covering adversarial robustness, safety gates, workflow quality, and operator-level checks for Vercel AI SDK AI products.
About Vercel AI SDK
Vercel AI SDK is the open-source TypeScript-first AI framework from Vercel — the `ai` npm package. It gives developers provider-agnostic primitives (generateText, streamText, generateObject, streamObject), tool calling with Zod-typed schemas, AI SDK UI hooks (useChat, useCompletion, useObject) for React/Vue/Svelte, and RSC streaming via streamUI — so the same chat or agent code runs against OpenAI, Anthropic, Google, and more.
How complete this published benchmark library is across datasets, metrics, rubrics, use-case maps, and pack context. This is library coverage, not an agent performance score.
Test datasets
8/8 packs
Scoring metrics
0/8 packs
Judge rubrics
8/8 packs
Use-case maps
0/8 packs
Pack context
8/8 packs
Available eval packs for Vercel AI SDK
8 packs ready to run.
Core Generate And Stream Text
Evaluates Vercel AI SDK's Core: generateText / streamText across 9 scenario-based test cases, each graded against an expected-behavior rubric by an LLM judge, from Corsac's AI SDK eval coverage.
Embeddings Image Speech
Evaluates Vercel AI SDK's Embeddings, Image & Speech across 9 scenario-based test cases, each graded against an expected-behavior rubric by an LLM judge, from Corsac's AI SDK eval coverage.
Middleware Telemetry Safety
Evaluates Vercel AI SDK's Middleware, Telemetry & Safety across 10 scenario-based test cases, each graded against an expected-behavior rubric by an LLM judge, from Corsac's AI SDK eval coverage.
Providers And Registry
Evaluates Vercel AI SDK's Providers & Provider Registry across 9 scenario-based test cases, each graded against an expected-behavior rubric by an LLM judge, from Corsac's AI SDK eval coverage.
Rsc Streaming Ui
Evaluates Vercel AI SDK's RSC Streaming & UI Generation across 9 scenario-based test cases, each graded against an expected-behavior rubric by an LLM judge, from Corsac's AI SDK eval coverage.
Structured Outputs
Evaluates Vercel AI SDK's Structured Outputs (generateObject / streamObject) across 9 scenario-based test cases, each graded against an expected-behavior rubric by an LLM judge, from Corsac's AI SDK eval coverage.
Tool Calling
Evaluates Vercel AI SDK's Tool Calling across 9 scenario-based test cases, each graded against an expected-behavior rubric by an LLM judge, from Corsac's AI SDK eval coverage.
Ui Hooks
Evaluates Vercel AI SDK's UI Hooks (React/Vue/Svelte) across 9 scenario-based test cases, each graded against an expected-behavior rubric by an LLM judge, from Corsac's AI SDK eval coverage.
Why eval Vercel AI SDK AI
Vercel AI SDK's AI features ship behind brand promises about accuracy, safety, and reliability. Buyers and integrators need to know those promises hold up under adversarial prompts, edge-case workflows, and the long tail of real customer inputs — not just the demo path.
The Corsac eval library for Vercel AI SDK measures four dimensions teams care about most when deploying code assistant agents:
- Adversarial robustness — does the agent resist prompt injection, jailbreaks, and social-engineering attempts?
- Workflow quality— does it complete the task buyers were shown in the demo, on inputs that don't look like the demo?
- Safety gates — does it escalate or refuse when it should, and only then?
- Operator quality — does it preserve analyst trust by surfacing the right context at the right time?
Every eval pack above is hand-authored against Vercel AI SDK's public product surface and runnable in Corsac with your own data.