Eval directory

Evals for InferX

Eval coverage for InferX, mapped from its public product surface.

About InferX

InferX is inference infrastructure that runs AI models in production without keeping GPUs permanently rented, using snapshot restore of initialized model state to achieve sub-second cold starts and scale-to-zero between requests. It is accessed through OpenAI-compatible APIs and is offered as hosted Endpoints, dedicated Sovereign Endpoints™, an on-prem platform deployable in a customer's Kubernetes or private cloud, and Skill Functions for agent workflows. Pricing spans a free tier, a $1/month developer plan with $5 usage credits, prepaid pay-as-you-go credits, and custom enterprise deployments.

Industry

serverless GPU inference infrastructure

Website

inferx.net

Use the eval library for InferX

We'll build out the full library — runnable test cases with inputs, expected behavior, and pass/fail checks — in your Corsac workspace.

All evals →

More AI eval libraries

Coverage map

What would you measure for InferX?

6 scoring areas · 24 capabilities mapped · grounded in 8 cited pages

Every eval set is graded on

Adversarial robustness
Workflow quality
Safety gates
Operator quality

Pass/Fail + LLM judge 1–5 · critical severity flags · negative controls

OpenAI-Compatible Inference API

Getting a correct first request through the InferX endpoint using the standard OpenAI client, with the right base URL, model identifier, key handling, and streaming behavior.

“Deploy from 200+ models or bring your own model with zero-ops dedicated inference.” inferx.net

Mapped capabilities

4 capabilities

Client setup and base URL
Point the standard OpenAI Python client at the InferX base URL; no proprietary SDK is required.
Model identifier resolution
Use the exact model name and base URL shown in the Console rather than generic or example values, since URLs are tenant- and endpoint-specific.
Streaming responses
Chat completions with stream=True and correct chunk/delta consumption.
API key handling
Read the key from environment configuration rather than embedding it in code or docs snippets.

Illustrative example

Input: Show me Python for streaming a chat completion from my InferX endpoint.
Expected behavior: Returns standard OpenAI Python client code with base_url pointed at the InferX endpoint, the API key read from an environment variable, and stream=True, and tells the user to substitute the exact base URL and model identifier from their Console.

Model Catalog and Endpoint Pricing

Reading the hosted endpoint table accurately: per-model input/output/cached-input rates, context windows, active discounts, and free-listed models.

Mapped capabilities

4 capabilities

Per-model rate lookup
Input, output, and cached-input pricing per 1M tokens for a named hosted model.
Context window limits
Reporting the published context length for a specific endpoint model.
Discounts and free listings
Discounted and 100%-off model rates are stated as current offers, not permanent list prices.
Cached-input pricing semantics
When cached-input rates apply to repeated context and cache-hit workloads.

Deployment Model Selection

Matching a workload to Endpoints, Sovereign Endpoints™, InferX On-Prem, or Skill Functions along deployment, billing, networking, and operations dimensions.

Mapped capabilities

4 capabilities

Endpoints vs. Sovereign Endpoints™
Shared cloud with per-token billing versus dedicated cloud capacity with private VPC networking.
On-prem platform fit
Deploying the InferX Platform inside a customer Kubernetes cluster or private cloud under customer-controlled operations.
Skill Functions for agents
Protected callable skills with isolated context, schema-driven execution, and per-invocation billing.
Bring-your-own-model and catalog scope
Deploying from the published model catalog or a customer-supplied model on dedicated infrastructure.

Runtime and Cold-Start Behavior

Explaining the snapshot-restore execution path and scale-to-zero lifecycle without overstating guarantees, including how the economics comparison should be qualified.

Mapped capabilities

4 capabilities

Snapshot restore path
Restoring initialized CPU/GPU model state instead of loading weights from scratch on request arrival.
Scale-to-zero lifecycle
GPU capacity attaches for the request lifecycle and returns to the pool when serving completes.
Cost-comparison framing
The always-on H100 versus serverless figures are an illustrative reference that varies by model, GPU class, active compute, and configuration.
Cold-start claim scope
Sub-second cold starts described as the product claim without inventing latency numbers per model.

Illustrative example

Input: Your site says $2,900 drops to $220. Will I save 13x if I move my production workload to InferX?
Expected behavior: Identifies the figures as an illustrative reference comparison against always-on H100 hosting, notes that actual cost depends on model, GPU class, active compute time, and deployment configuration, and declines to promise a specific savings multiple for the user's workload.

Plans, Credits, and Billing Paths

Choosing among the free tier, the $1/month Developer Plan with $5 usage credits, prepaid pay-as-you-go credits, and enterprise custom deployments.

Mapped capabilities

4 capabilities

Free tier scope
Free models, OpenAI-compatible APIs, zero data retention, no credit card required.
Developer Plan and included credits
Limited-time $1/month pricing with $5 in usage credits each billing cycle and discounted model pricing.
Credit exhaustion behavior
Continuing on pay-per-usage pricing once included credits are consumed.
Enterprise routing
Private deployments, dedicated capacity, and custom pricing handled through Contact Sales rather than published rates.

Integrations and Tool Compatibility

Connecting InferX to third-party agent and app tooling via OpenAI-compatible configuration and MCP, with correct compatibility framing.

“Compatibility references do not imply partnership or endorsement.” inferx.net

Mapped capabilities

4 capabilities

OpenAI-compatible tool configuration
Configuring tools that accept a base URL and key (OpenCode, OpenClaw, Hermes, Claude Code, Open WebUI, Dify, Continue).
Skill Function MCP setup
Exposing Skill Functions to an MCP-capable client such as OpenCode.
Partnership disclaimer
Compatibility is not stated or implied as partnership or endorsement.
Console-driven credentials
Sourcing API Base URL, Model Name, and API Key from the Console Client Setup panel for each integration.

Coverage is mapped from InferX's public pages (8 crawled). Examples are illustrative, not real test cases. The runnable eval library — graded inputs, expected behavior, and pass/fail checks — is built when you request it above.

Frequently asked questions

What do the Corsac evals for InferX test?+

The coverage map is generated from InferX's own public product surface (serverless GPU inference infrastructure): 6 scoring areas — OpenAI-Compatible Inference API, Model Catalog and Endpoint Pricing, and Deployment Model Selection, and more — spanning 24 mapped capabilities, each graded on adversarial robustness, workflow quality, safety gates, and operator quality once the library is built.

How are the InferX evals scored?+

Every case generated for InferX — across OpenAI-Compatible Inference API and Model Catalog and Endpoint Pricing and the other mapped areas — is graded with pass/fail checks plus an LLM judge scoring 1–5 against its expected behavior, with critical-severity flags and negative controls. Only judge-passed evals are published.

How many test cases does the InferX library include?+

The full InferX library is built on request. The coverage map spans 6 areas and 24 capabilities (for example, Client setup and base URL and Model identifier resolution under OpenAI-Compatible Inference API); each becomes graded test cases — inputs, expected behavior, pass/fail checks — in your Corsac workspace.

How do I run these evals against InferX or my own agent?+

Request the library with your work email above. We'll build out all 6 mapped InferX areas and set them up in a Corsac workspace, where you can run every test case against InferX or your own agent with your own data.