Eval directory · AI Platform

Evals for Featherless

Eval coverage for Featherless, mapped from its public product surface.

About Featherless

Featherless AI is a serverless inference platform that provides API access to a large catalog of open-weight language models (Qwen, Llama, Mistral, DeepSeek, GLM, and others) through a single OpenAI-compatible API key. It also offers dedicated GPU deployments, including AMD MI325X instances for models like GLM 5.2, positioned as a lower-cost alternative to proprietary frontier models for always-on agent workloads. Documentation covers tool calling, vision, embeddings, and agent integrations.

Industry

serverless LLM inference / open-model hosting platform

Website

featherless.ai

Use the eval library for Featherless

We'll build out the full library — runnable test cases with inputs, expected behavior, and pass/fail checks — in your Corsac workspace.

Related in AI Platform

All evals →

More AI Platform eval libraries

Coverage map

What would you measure for Featherless?

6 scoring areas · 24 capabilities mapped · grounded in 8 cited pages

Every eval set is graded on

Adversarial robustness
Workflow quality
Safety gates
Operator quality

Pass/Fail + LLM judge 1–5 · critical severity flags · negative controls

Model Catalog & Selection

Helping a user navigate a 40,000+ model catalog and pick a model by size, family, and stated use case (coding/agents, reasoning, roleplay and creative writing, small models).

Mapped capabilities

4 capabilities

Model lookup by exact repo id
Resolving strings like Qwen/Qwen3-Coder-30B-A3B-Instruct or zai-org/GLM-5.2 without altering or inventing ids.
Use-case-driven recommendation
Mapping a stated workload to the catalog categories surfaced on the site (productivity, reasoning, RP & creative writing, small models).
Model compatibility constraints
Explaining which model types are supported for serverless inference per the models & compatibility docs.
Catalog scope claims
Stating catalog breadth and model availability without overstating support for unlisted modalities.

OpenAI-Compatible Inference API

Correctness of API guidance: base URL, auth, endpoint selection, and the documented request/response surface.

“Our API interface is OpenAI compatible” featherless.ai

Mapped capabilities

4 capabilities

OpenAI SDK drop-in configuration
base_url https://api.featherless.ai/v1 plus a Featherless API key, with no other client changes.
Endpoint selection and options
Choosing among /v1/chat/completions, /v1/completions, /v1/models, /v1/tokenize, and /v1/plan for a given task.
Direct HTTP request construction
Headers, JSON body, and model field for non-SDK integrations.
Error codes and retry guidance
Interpreting documented error codes and advising recovery rather than guessing at causes.

Illustrative example

Input: I already use the OpenAI Python SDK. What is the smallest change needed to call Qwen/Qwen2.5-7B-Instruct on Featherless instead of OpenAI?
Expected behavior: Says to keep the same SDK and chat.completions.create call, setting base_url to https://api.featherless.ai/v1 and api_key to a Featherless key, with model set to the exact id Qwen/Qwen2.5-7B-Instruct.

Tool Calling, Vision & Embeddings

Build-with-Featherless capabilities documented beyond plain text completion.

Mapped capabilities

4 capabilities

Tool/function calling setup
Declaring tools and handling tool-call responses through the OpenAI-compatible schema.
Vision requests
Sending image inputs to models documented as vision-capable.
Embeddings usage
Calling the embeddings surface and distinguishing it from chat completions.
Chat template kwargs
Passing template arguments documented for open-weight chat models.

Plans, Credits & Concurrency

Account-side rules a developer hits in production: plan tiers, request pricing and credits, and concurrent unit limits.

Mapped capabilities

4 capabilities

Plan and credit explanation
Describing request pricing and credit consumption as documented, without inventing tiers or rates.
Concurrent unit limits
Explaining how concurrency units are counted and what happens at the limit.
Concurrency stream endpoint
Using /account/concurrency/stream to observe live capacity.
Throttling recovery advice
What a client should do when concurrency or rate limits are reached.

Dedicated GPU Deployments & Cost Economics

Dedicated instances (including AMD MI325X hosting GLM 5.2) positioned for always-on agent workloads, plus the cost-comparison claims used to justify them.

“Featherless AI dedicated GLM 5.2 instances start at $7,592 per month on four AMD MI325X GPUs.” featherless.ai

Mapped capabilities

4 capabilities

Dedicated instance configuration
Hardware, model, and starting price as published for dedicated GLM 5.2.
Published throughput figures
Single-stream and aggregate tok/s numbers, including the FP8 variant, quoted only as published.
Serverless vs dedicated fit
When a sustained agent workload justifies dedicated capacity over serverless.
Cost-claim discipline
Repeating the telco cost-reduction case study with its stated scope and hedges, not as a general guarantee.

Illustrative example

Input: What does a dedicated GLM 5.2 instance cost per month on Featherless, and what hardware does it run on?
Expected behavior: States that dedicated GLM 5.2 instances start at $7,592 per month on four AMD MI325X GPUs, and does not invent additional tiers, discounts, or contract terms beyond that published starting figure.

Agent Integrations & Client Migration

Getting third-party agent and chat clients pointed at Featherless, and answering the trust questions that come up during migration.

Mapped capabilities

4 capabilities

Documented agent clients
Setup guidance for the integrations listed in docs (Open WebUI, SillyTavern, Hermes Agent, Claw-family agents, Space Bot).
Migration from other providers
Reconfiguring an OpenAI- or OpenRouter-based app with minimal changes.
Privacy and logging answers
Representing the documented privacy and logging position accurately.
Status and reliability signals
Pointing to the status surface when a user reports availability problems.

Coverage is mapped from Featherless's public pages (8 crawled). Examples are illustrative, not real test cases. The runnable eval library — graded inputs, expected behavior, and pass/fail checks — is built when you request it above.

Frequently asked questions

What do the Corsac evals for Featherless test?+

The coverage map is generated from Featherless's own public product surface (serverless LLM inference / open-model hosting platform): 6 scoring areas — Model Catalog & Selection, OpenAI-Compatible Inference API, and Tool Calling, Vision & Embeddings, and more — spanning 24 mapped capabilities, each graded on adversarial robustness, workflow quality, safety gates, and operator quality once the library is built.

How are the Featherless evals scored?+

Every case generated for Featherless — across Model Catalog & Selection and OpenAI-Compatible Inference API and the other mapped areas — is graded with pass/fail checks plus an LLM judge scoring 1–5 against its expected behavior, with critical-severity flags and negative controls. Only judge-passed evals are published.

How many test cases does the Featherless library include?+

The full Featherless library is built on request. The coverage map spans 6 areas and 24 capabilities (for example, Model lookup by exact repo id and Use-case-driven recommendation under Model Catalog & Selection); each becomes graded test cases — inputs, expected behavior, pass/fail checks — in your Corsac workspace.

How do I run these evals against Featherless or my own agent?+

Request the library with your work email above. We'll build out all 6 mapped Featherless areas and set them up in a Corsac workspace, where you can run every test case against Featherless or your own agent with your own data.