Eval directory

Evals for BLACKBOX AI

Eval coverage for BLACKBOX AI, mapped from its public product surface.

About BLACKBOX AI

Blackbox is an inference platform that serves open- and closed-weight models through a single OpenAI-compatible endpoint (the Blackbox Router) alongside dedicated single-tenant Enterprise Inference deployments. It markets end-to-end encrypted inference, gateway-enforced zero data retention and no training, and 300+ models under one key, bill, and dashboard. It also ships agent surfaces — an Agents API, VS Code extension, and CLI — on the same per-token commit, priced pay-as-you-go or via annual committed enterprise spend.

Industry

enterprise LLM inference and model routing platform

Website

www.blackbox.ai

Use the eval library for BLACKBOX AI

We'll build out the full library — runnable test cases with inputs, expected behavior, and pass/fail checks — in your Corsac workspace.

All evals →

More AI eval libraries

Coverage map

What would you measure for BLACKBOX AI?

6 scoring areas · 23 capabilities mapped · grounded in 8 cited pages

Every eval set is graded on

Adversarial robustness
Workflow quality
Safety gates
Operator quality

Pass/Fail + LLM judge 1–5 · critical severity flags · negative controls

Router API compatibility and model access

The Blackbox Router as a drop-in OpenAI-compatible endpoint: base URL and single API key replacing per-provider keys, namespaced model strings (blackboxai/<vendor>/<model>), streaming, and one-string switching across the 300+ open and closed catalog.

“We suppress retention and training on routed traffic through provider terms and per-request flags” www.blackbox.ai

Mapped capabilities

4 capabilities

OpenAI-compatible request shape and base-URL/key swap
Answers about pointing an existing OpenAI SDK or curl call at https://enterprise.blackbox.ai/v1 with BLACKBOX_API_KEY, replacing OPENAI/ANTHROPIC/GOOGLE keys.
Model identifier syntax and provider switching
Correct namespaced model strings and the claim that changing frontiers means changing one string, not the request format.
Streaming and per-request options
Behavior of stream:true and standard chat-completions parameters through the router.
Catalog breadth and vendor coverage
Which model families are described as reachable (Claude, GPT, Gemini, Grok, Llama, Mistral, DeepSeek, Qwen, Nemotron) without asserting specific unlisted models.

Illustrative example

Input: I have a working curl call to your endpoint using blackboxai/nvidia/nemotron-3-ultra. What do I change to send the same prompt to Claude Opus 4.8 instead?
Expected behavior: States that only the model string changes — to blackboxai/anthropic/claude-opus-4.8 — while the base URL, single Blackbox API key, and OpenAI-compatible request body stay identical. Does not introduce a second provider key or a different endpoint.

Privacy, retention, and data-handling policy

The platform's central trust claims and their stated limits: zero data retention and no training enforced at the gateway, end-to-end encrypted inference, and PII removal before prompts reach closed models on Enterprise.

“The gateway enforces zero data retention and no training.” www.blackbox.ai

Mapped capabilities

4 capabilities

Zero retention and no-training enforcement point
That enforcement is described as occurring at the gateway, in memory and discarded, and is contractual plus DPA on Enterprise.
Stated qualifiers on routed traffic
That retention/training suppression on routed traffic is via provider terms and per-request flags wherever the provider API supports it — a qualified, not absolute, claim.
PII removal before closed models
The Enterprise-tier behavior where the downstream provider sees the task, not the person it is about.
End-to-end encryption scope
Encryption described as covering every connection and every plan, including pay-as-you-go.

Illustrative example

Input: If I route a prompt through the Router to a third-party closed model, is that prompt guaranteed never to be retained or trained on by that provider?
Expected behavior: Explains that zero retention and no training are enforced at the Blackbox gateway, but that suppression on traffic routed onward is achieved through provider terms and per-request flags wherever the provider API supports it — a qualified rather than absolute guarantee. May note that Enterprise adds PII removal before closed models plus contractual and DPA commitments.

Enterprise Inference and tenancy controls

Dedicated single-tenant deployments of a customer-chosen open-weight model on reserved Blackbox GPUs, plus the enterprise governance surface listed alongside it.

“On Enterprise, we remove PII before prompts reach a closed model” www.blackbox.ai

Mapped capabilities

4 capabilities

Single-tenant isolation and reserved capacity
No shared pools, no other customers, customer-selected open-weight model, verified top speed.
Identity and access governance
SAML SSO, SCIM, RBAC, and audit logs as Enterprise-tier features; SSO sign-in across every surface.
Residency, keys, and contractual terms
Data residency, customer-managed keys, custom contracts, custom SLAs.
Capacity and rate-limit guarantees
Guaranteed TPM and custom rate limits versus standard rate limits on pay-as-you-go.

Routing reliability and failover

How the routing fabric behaves when an upstream provider degrades — the smart routing, failover, and caching listed on every plan, and the timeout path depicted in the routing fabric.

Mapped capabilities

3 capabilities

Failover across upstream providers
Behavior when a routed provider times out or is unavailable, per the smart routing and failover feature listing.
Caching behavior
Caching as a listed platform feature available on all plans.
Rate-limit tiers as a degradation surface
Standard limits on pay-as-you-go versus guaranteed TPM on Enterprise.

Agent surfaces and developer tooling

Agents API, VS Code extension, CLI, IDE, mobile, and Builder — all drawing on one per-token commit, working with existing agent harnesses, with dedicated Remote Agent runners on Enterprise.

Mapped capabilities

4 capabilities

Shared commit across every surface
That CLI, VS Code, IDE, mobile, and Builder consume the same per-token balance rather than separate seats or fees.
Harness interoperability
Works-with-every-agent-harness claim, including pointing a third-party harness such as OpenClaw at a Blackbox endpoint.
Multi-agent orchestration
/multi-agent and Chairman LLM orchestration as listed plan features.
Remote Agent runners
Dedicated Remote Agent runners with SSO on the Enterprise plan.

Pricing, commitments, and performance claims

Commercial mechanics — rack rate versus annual committed spend with burn-down, discount tiers, absence of platform fees or seats — together with the disciplined restatement of published speed and benchmark results.

Mapped capabilities

4 capabilities

Plan mechanics and commit discounts
Prepaid metered credits at rack rate versus annual PO burn-down; open and closed model discount tiers grow with committed spend.
Fee structure
No platform fees, no markups, no seats; implementation included at $0 on Enterprise.
Speed and price comparisons
Artificial Analysis July 2026 snapshot figures for Nemotron 3 Ultra and the terms under which they are comparable.
Benchmark claim precision
Terminal-Bench v2.1 AA-basis versus as-run figures, and the stated caveat that quality comparisons support preserved quality rather than proving score differences.

Coverage is mapped from BLACKBOX AI's public pages (8 crawled). Examples are illustrative, not real test cases. The runnable eval library — graded inputs, expected behavior, and pass/fail checks — is built when you request it above.

Frequently asked questions

What do the Corsac evals for BLACKBOX AI test?+

The coverage map is generated from BLACKBOX AI's own public product surface (enterprise LLM inference and model routing platform): 6 scoring areas — Router API compatibility and model access, Privacy, retention, and data-handling policy, and Enterprise Inference and tenancy controls, and more — spanning 23 mapped capabilities, each graded on adversarial robustness, workflow quality, safety gates, and operator quality once the library is built.

How are the BLACKBOX AI evals scored?+

Every case generated for BLACKBOX AI — across Router API compatibility and model access and Privacy, retention, and data-handling policy and the other mapped areas — is graded with pass/fail checks plus an LLM judge scoring 1–5 against its expected behavior, with critical-severity flags and negative controls. Only judge-passed evals are published.

How many test cases does the BLACKBOX AI library include?+

The full BLACKBOX AI library is built on request. The coverage map spans 6 areas and 23 capabilities (for example, OpenAI-compatible request shape and base-URL/key swap and Model identifier syntax and provider switching under Router API compatibility and model access); each becomes graded test cases — inputs, expected behavior, pass/fail checks — in your Corsac workspace.

How do I run these evals against BLACKBOX AI or my own agent?+

Request the library with your work email above. We'll build out all 6 mapped BLACKBOX AI areas and set them up in a Corsac workspace, where you can run every test case against BLACKBOX AI or your own agent with your own data.