Eval Library
Vercel AI SDK
For Vercel AI SDKCode AssistantAi Platform

Providers And Registry

Vercel AI SDK (`ai` npm package) · Vercel AI SDK

AI SDK — Vercel AI SDK

Vercel AI SDK evals — Providers & Provider Registry (relift v3 InfraRed)

About Vercel AI SDK

Vercel AI SDK is the open-source TypeScript-first AI framework from Vercel — the `ai` npm package. It gives developers provider-agnostic primitives (generateText, streamText, generateObject, streamObject), tool calling with Zod-typed schemas, AI SDK UI hooks (useChat, useCompletion, useObject) for React/Vue/Svelte, and RSC streaming via streamUI — so the same chat or agent code runs against OpenAI, Anthropic, Google, and more.

Employees

~500

Industry

AI Framework / SDK

Headquarters

San Francisco, CA

Website

ai-sdk.dev

Sample tests· showing 3 of 9

#InputExpected behaviorCheck
01

Operator imports `import { anthropic } from '@ai-sdk/anthropic'` and ships; production logs show MODULE_NOT_FOUND because only `ai` was installed.

Each provider is a separate npm package: `npm install @ai-sdk/anthropic` (or openai/google/mistral/azure). The `ai` core package only ships the interface; provider plugins implement LanguageModelV2. Pin compatible versions (provider plugin major must match `ai` major).

Pass / FailAi Platformhigh
02

Operator wants to use a self-hosted OpenAI-compatible endpoint (e.g., a local llama.cpp server) via @ai-sdk/openai.

Use `createOpenAI({ baseURL: 'http://localhost:8080/v1', apiKey: '...' })` to produce a provider instance pointing at the custom endpoint. Do not mutate globals or set OPENAI_API_BASE env (that affects every importer). Pass the instance explicitly to generateText({ model: myOpenAI('gpt-4o') }).

Pass / FailAi Platformmedium
03

Operator builds a custom provider that aliases short names to specific model ids and wants unaliased ids to pass through to the default openai provider.

Use `customProvider({ languageModels: { 'fast': openai('gpt-4o-mini'), 'deep': anthropic('claude-opus-4-5') }, fallbackProvider: openai })`. Unmatched ids resolve via fallbackProvider. Without fallbackProvider, unmatched ids throw NoSuchModelError.

Pass / FailAi Platformmedium

How this eval is graded

Grade against expected.ideal_behavior and expected.rubric. Per-criterion pass requires mean >= 4.0 and no criterion below 3.

Rubric criteria

  • Vercel Ai Sdk
  • Ai Platform
  • Providers And Registry

Recommended for

Vercel AI SDK (`ai` npm package)Vercel AI SDK customers

Works with

Related evals

Run this eval in your workspace

Connect your data, configure thresholds, and review results with your team.