Eval directory

Evals for Sapiom

Eval coverage for Sapiom, mapped from its public product surface.

About Sapiom

Sapiom is production infrastructure for building, running, and scaling AI agents, comprising three products: Router, agent.studio, and Runtime. Router exposes an OpenAI-compatible chat-completions endpoint that selects an allowed model per request under cost and policy constraints, while Runtime deploys typed step graphs that can pause on signals, resume at named steps, and retry within a bounded attempt budget. agent.studio combines AI chat, Canvas, Steps, and ordinary TypeScript in one workspace for authoring and deploying agents, with pricing from a free Developer tier up to a custom Scale tier.

Industry

AI agent infrastructure platform

Website

www.sapiom.ai

Use the eval library for Sapiom

We'll build out the full library — runnable test cases with inputs, expected behavior, and pass/fail checks — in your Corsac workspace.

All evals →

More AI eval libraries

Coverage map

What would you measure for Sapiom?

6 scoring areas · 24 capabilities mapped · grounded in 8 cited pages

Every eval set is graded on

Adversarial robustness
Workflow quality
Safety gates
Operator quality

Pass/Fail + LLM judge 1–5 · critical severity flags · negative controls

Router request compatibility and model selection

Whether an OpenAI-compatible agent can switch to the Sapiom chat-completions endpoint without changing its request shape, and how Router picks a model per request from task shape, context size, latency, quality, cost, reliability, and availability signals.

“Sapiom selects the most efficient allowed model for each task and meters every call.” www.sapiom.ai

Mapped capabilities

4 capabilities

Endpoint and auth swap
POST to the Sapiom chat-completions path with a Bearer API key and JSON content type; what the caller must change versus keep.
OpenAI-compatible request shape
Standard messages payload continues to work; behavior when the caller omits or specifies fields the routing layer would otherwise decide.
Per-request selection signals
Selection across efficient, mid-tier, and frontier tiers using task shape, input and context size, latency, quality, and cost.
Route alternatives under pressure
Retry route, fallback route, and policy hold as distinct outcomes when the preferred allowed model is unavailable or disallowed.

Illustrative example

Input: Our agent already speaks the OpenAI chat-completions format. We point it at the Sapiom Router URL with our Bearer key and send no model field. What runs, and does our client code change?
Expected behavior: Explains that Router evaluates each request at execution time and selects the most efficient allowed model under the account's cost and policy constraints, and that the OpenAI-compatible message format is unchanged, so only the endpoint URL and API key differ.

Policy, allowed models, and governance

How company policy narrows what Router may select, and what governance surfaces exist for larger fleets: account and per-agent rules, audit trail, org-wide policy, SSO, and telemetry export.

“Org-wide policies, audit trail, and SSO” www.sapiom.ai

Mapped capabilities

4 capabilities

Allowed-model constraints
Company policy and the allowed-model set bounding selection; policy hold as the outcome when no allowed path fits.
Account and per-agent rules
Account-level spending rules on Developer; per-agent rules across a fleet on Startup.
Org-wide policy and audit trail
Org-wide policies, audit trail, and SSO described as Scale-tier controls.
Multi-tenant and telemetry export
Telemetry export and multi-tenant management for teams running many agents.

Cost metering, receipts, and plan economics

Per-call and per-run metering, cost breakdowns and receipts, spending limits, and how included runs and overage rates differ across Developer, Startup, and Scale.

Mapped capabilities

4 capabilities

Per-call and per-run metering
Every Router call metered; runs listed with service, cost, and status in the run view.
Included runs and overage
Developer included runs with per-run overage, Startup monthly runs with a lower overage rate, Scale custom volume pricing.
Cost breakdown and receipts
Run cost breakdowns, spending limits, and exportable receipts.
What does not count
Platform failures and automatic retries excluded from counted runs; metered capabilities billed per use.

Runtime step graph and directives

Typed step graphs where each step names its allowed destinations and returns a directive that fully describes its result, making every transition explicit and inspectable.

“Deploy typed step graphs with explicit retries, signals, schedules, and per-attempt traces.” www.sapiom.ai

Mapped capabilities

4 capabilities

Directive semantics
goto, terminate, fail, retry, and pauseUntilSignal as the complete set of step outcomes.
Declared destinations
Each step naming its allowed next steps; the runtime following the written graph rather than inferring a path.
Typed agent definition
defineAgent and the @sapiom/agent authoring surface for typed steps and inputs.
Local check and run
CLI check and local run before deployment, using the same graph that runs in the cloud.

Illustrative example

Input: A step returns retry with a 1000ms delay while ctx.attempts + 1 is under 3, and otherwise returns fail. On the first attempt the step throws an unexpected exception. Does that consume budget?
Expected behavior: Explains that unexpected errors and explicit retry directives share the same bounded attempt budget, so the thrown exception consumes an attempt against the cap. Notes that the attempt count is zero-based and that returning fail stops the run as a known terminal case.

Durable pause, resume, and scheduling

Executions that suspend on a declared signal and resume at a named step when the payload arrives, plus scheduled runs and per-attempt trace inspection.

“Deploy typed agents that pause for signals, resume at named steps, and stay inspectable” www.sapiom.ai

Mapped capabilities

4 capabilities

pauseUntilSignal and resume step
A step declaring its signal and resume step; execution suspended until the signal is delivered.
Signal payload delivery
Result payload routed to the declared resume step, scoped by execution identifier.
Run inspection and traces
Inspector states such as waiting, completed, and per-attempt traces across a run.
Scheduled executions
Recurring or scheduled runs via the CLI schedule surface.

agent.studio authoring and ship loop

One workspace carrying an agent from business intent to a deployed cloud run: AI chat, Canvas, Steps, and ordinary TypeScript on the same project, plus templates and the link-deploy-run-inspect transitions.

“The code remains ordinary TypeScript in a project you can inspect.” www.sapiom.ai

Mapped capabilities

4 capabilities

Intent to typed agent
Turning a described outcome and its approval points into a typed agent inspectable before anything runs.
Three views of one agent
Canvas route through the graph, Steps as a scan-ready list, and Code as the generated TypeScript.
Source ownership
Opening the project tree and copying step source without leaving the workspace.
Link, deploy, run, inspect
Linking a project, deploying its current commit, starting a cloud execution, and inspecting the build or run as explicit transitions.

Coverage is mapped from Sapiom's public pages (8 crawled). Examples are illustrative, not real test cases. The runnable eval library — graded inputs, expected behavior, and pass/fail checks — is built when you request it above.

Frequently asked questions

What do the Corsac evals for Sapiom test?+

The coverage map is generated from Sapiom's own public product surface (AI agent infrastructure platform): 6 scoring areas — Router request compatibility and model selection, Policy, allowed models, and governance, and Cost metering, receipts, and plan economics, and more — spanning 24 mapped capabilities, each graded on adversarial robustness, workflow quality, safety gates, and operator quality once the library is built.

How are the Sapiom evals scored?+

Every case generated for Sapiom — across Router request compatibility and model selection and Policy, allowed models, and governance and the other mapped areas — is graded with pass/fail checks plus an LLM judge scoring 1–5 against its expected behavior, with critical-severity flags and negative controls. Only judge-passed evals are published.

How many test cases does the Sapiom library include?+

The full Sapiom library is built on request. The coverage map spans 6 areas and 24 capabilities (for example, Endpoint and auth swap and OpenAI-compatible request shape under Router request compatibility and model selection); each becomes graded test cases — inputs, expected behavior, pass/fail checks — in your Corsac workspace.

How do I run these evals against Sapiom or my own agent?+

Request the library with your work email above. We'll build out all 6 mapped Sapiom areas and set them up in a Corsac workspace, where you can run every test case against Sapiom or your own agent with your own data.