Eval directory

Evals for assistant-ui

Eval coverage for assistant-ui, mapped from its public product surface.

About assistant-ui

assistant-ui is an open-source TypeScript/React library of primitives and components for building ChatGPT-style AI chat interfaces. It handles streaming, multi-turn state, tool calls, and attachments while letting developers bring their own backend via runtimes for Vercel AI SDK, LangGraph, LangChain, and others. A companion hosted product, assistant-cloud, offers a managed backend with chat history and thread management on free, Pro, and Enterprise tiers, and the library also targets React Native and terminal (Ink) UIs.

Industry

AI chat UI component library (React)

Website

www.assistant-ui.com

Use the eval library for assistant-ui

We'll build out the full library — runnable test cases with inputs, expected behavior, and pass/fail checks — in your Corsac workspace.

All evals →

More AI eval libraries

Coverage map

What would you measure for assistant-ui?

6 scoring areas · 24 capabilities mapped · grounded in 8 cited pages

Every eval set is graded on

Adversarial robustness
Workflow quality
Safety gates
Operator quality

Pass/Fail + LLM judge 1–5 · critical severity flags · negative controls

Runtimes & Backend Integration

assistant-ui is bring-your-own-backend: a runtime connects the components to an AI backend. Covers choosing and wiring the documented runtimes and the custom-server path.

Mapped capabilities

4 capabilities

Runtime selection for an existing stack
Mapping a stated backend (Vercel AI SDK, LangGraph, LangChain, Google ADK, AG-UI, A2A, OpenCode) to the matching documented runtime.
Custom / own-server runtime
Guidance when no first-party runtime matches and the developer supplies their own server.
Runtime comparison
Explaining tradeoffs across runtimes without inventing integrations absent from the docs.
Project setup entrypoints
npx assistant-ui init, npx assistant-ui@latest create, and adding to an existing app.

Illustrative example

Input: I already run a LangGraph agent and want a React chat UI without writing my own backend adapter. Which assistant-ui runtime should I use?
Expected behavior: Recommends the documented LangGraph runtime and notes assistant-ui is bring-your-own-backend, with a runtime per stack. Does not propose a custom server or an integration not listed in the docs.

Chat UI Primitives & Components

Unstyled, composable primitives (Thread, Composer, Message) and production-ready components with theming and sensible defaults, including shadcn/ui-flavored setups.

“React primitives for building AI chat interfaces.” www.assistant-ui.com

Mapped capabilities

4 capabilities

Primitive composition
Assembling Thread, Composer, and Message primitives into a working chat surface.
Theming and presets
Applying the documented looks (Base, ChatGPT, Claude, Grok, Gemini, Perplexity) and custom theming.
Components vs. primitives choice
When to drop in prebuilt components versus composing unstyled primitives.
API reference navigation
Locating the correct documented prop or primitive for a stated UI requirement.

Streaming & Conversation State

State management for live AI responses: streaming output, interruptions, retries, multi-turn threads, and branching conversations.

“Streaming, interruptions, retries, and multi-turn conversations.” www.assistant-ui.com

Mapped capabilities

4 capabilities

Streaming rendering
Handling incremental token streams and scroll/scroll-to-bottom behavior during generation.
Interruption and retry
Stopping an in-flight response and regenerating a turn.
Multi-turn thread state
Maintaining conversation context across turns within a thread.
Branching
Editing a message and navigating alternate conversation branches.

Tool Calls & Attachments

Rendering and controlling agent tool invocations and user-supplied files inside the chat surface.

Mapped capabilities

4 capabilities

Tool UI renderers
Registering tools with execute functions and custom per-tool UI.
ToolFallback behavior
Default rendering of unregistered tool calls, including expandable output and spinners.
Attachments
Adding and displaying file attachments in the composer per the attachments guide.
MCP server integration
Using the documented MCP server surfaces, including the assistant-ui docs MCP server.

Multi-Platform Targets

The same runtime and component model across web React, React Native, and terminal UIs via React Ink, including markdown rendering per platform.

“Rich markdown rendering with syntax highlighting, tables, and links” www.assistant-ui.com

Mapped capabilities

4 capabilities

React Ink terminal apps
Creating terminal chat apps (create --ink) with Ink-native primitives.
Terminal markdown rendering
Rich markdown as ANSI output with syntax highlighting, tables, and links.
React Native target
Building the mobile chat surface on the shared component model.
Runtime code sharing and migration
Reusing runtime setup, tool definitions, and model adapters across web, mobile, and terminal.

assistant-cloud: Persistence & Plans

The hosted managed backend offering chat history and thread management, and its Free / Pro / Enterprise commercial terms.

Mapped capabilities

4 capabilities

Chat history and persistence
Thread persistence across refreshes and thread management via the hosted backend.
Tier and MAU pricing
Free (200 MAU), Pro ($50/mo, 500 MAU, $0.10 per additional MAU), Enterprise custom pricing.
Enterprise capabilities
Own-backend integration, data replication, dedicated support, 99.99% uptime SLA, on-premises deployment.
Open source vs. cloud boundary
Distinguishing the MIT-licensed free library from the paid hosted backend.

Illustrative example

Input: We expect about 900 monthly users who each send at least one message. What would assistant-cloud Pro cost us per month?
Expected behavior: States Pro is $50/mo including 500 MAU, and that the 400 additional MAU bill at $0.10 each, giving $90/mo total. Does not quote a tier or rate absent from the pricing page.

Coverage is mapped from assistant-ui's public pages (8 crawled). Examples are illustrative, not real test cases. The runnable eval library — graded inputs, expected behavior, and pass/fail checks — is built when you request it above.

Frequently asked questions

What do the Corsac evals for assistant-ui test?+

The coverage map is generated from assistant-ui's own public product surface (AI chat UI component library (React)): 6 scoring areas — Runtimes & Backend Integration, Chat UI Primitives & Components, and Streaming & Conversation State, and more — spanning 24 mapped capabilities, each graded on adversarial robustness, workflow quality, safety gates, and operator quality once the library is built.

How are the assistant-ui evals scored?+

Every case generated for assistant-ui — across Runtimes & Backend Integration and Chat UI Primitives & Components and the other mapped areas — is graded with pass/fail checks plus an LLM judge scoring 1–5 against its expected behavior, with critical-severity flags and negative controls. Only judge-passed evals are published.

How many test cases does the assistant-ui library include?+

The full assistant-ui library is built on request. The coverage map spans 6 areas and 24 capabilities (for example, Runtime selection for an existing stack and Custom / own-server runtime under Runtimes & Backend Integration); each becomes graded test cases — inputs, expected behavior, pass/fail checks — in your Corsac workspace.

How do I run these evals against assistant-ui or my own agent?+

Request the library with your work email above. We'll build out all 6 mapped assistant-ui areas and set them up in a Corsac workspace, where you can run every test case against assistant-ui or your own agent with your own data.