Corsac
Use Corsac
All evals
Inferact

Eval directory · Medical & Clinical AI

Evals for Inferact

Evaluation packs covering adversarial robustness, safety gates, workflow quality, and operator-level checks for Inferact AI products.

About Inferact

Inferact was founded by vLLM creators and maintainers to grow the open-source inference engine and build commercial inference infrastructure.

Industry

AI Inference Infrastructure

Website

inferact.ai

Use the eval library for Inferact

All 42 test cases — inputs, expected behavior, and pass/fail checks — runnable in Corsac with your own data.

Generate your own →

Related in Medical & Clinical AI

All evals →

More Medical & Clinical AI eval libraries

Coverage map

What would you measure for Inferact?

3 areas · 42 graded scenarios

Every eval set is graded on

  • Adversarial robustness
  • Workflow quality
  • Safety gates
  • Operator quality

Pass/Fail + LLM judge 1–5 · critical severity flags · negative controls

01

Continuous Batching And Pagedattention Kv Cache Behavior

Mapped capabilities

22 scenarios

  • Mid-batch request admission

Public sample case

Input
Engineer notices intermittent CUDA OOM crashes on Inferact's serverless tier under bursty mixed-length traffic (many 1-token classification calls mixed with 4K-token generations) and suspects the per-step scheduler computes the t…
Expected behavior
Agent locates the scheduler's per-step admission loop, identifies that the running token-budget accumulator must be updated immediately after each admission decision (full prefill, chunked prefill, or decode token) rather than compared against a snapshot taken at step start, verifies the fix by tra…
Check
Pass / fail check

02

Openai Compatible Api Conformance

Mapped capabilities

8 scenarios

  • Chat completions SSE streaming conformance

Public sample case

Input
An agent's task runner calls Inferact's /v1/chat/completions with stream=true to summarize a document. Mid-stream, due to a backend logging hook, one SSE event is malformed: the JSON payload contains an unescaped literal newline …
Expected behavior
On hitting the JSON decode error for that event, the agent treats the stream as failed/corrupted at that point: it stops accumulating content past the bad event, raises/logs a clear parse error (e.g. 'SSE chunk failed JSON decode at event N'), and does not append a partial or hallucinated summary t…
Check
Pass / fail check

03

Quantization Lora And Multimodal Serving

Mapped capabilities

12 scenarios

  • FP8 quantized output correctness

Public sample case

Input
The engineer is about to switch the model backing a multi-step trade-sizing and risk-check agent from FP16 to FP8 on Inferact to cut serving cost. They ask the coding/ops agent to validate that quality is preserved before flippin…
Expected behavior
Agent designs a validation plan that explicitly separates reasoning/arithmetic/multi-step-logic prompts from generic conversational prompts, since FP8's narrow dynamic range degrades these unevenly; it specifies a held-out set of multi-step numeric/decision prompts representative of the trade-sizin…
Check
Pass / fail check

Frequently asked questions

What do the Corsac evals for Inferact test?+

Each eval pack tests Inferact's public product surface — including Continuous Batching And Pagedattention Kv Cache Behavior, Openai Compatible Api Conformance, and Quantization Lora And Multimodal Serving — against graded scenarios covering adversarial robustness, workflow quality, safety gates, and operator quality. Every pack is runnable in Corsac with your own data.

How are the Inferact evals scored?+

Pass/fail checks plus an LLM judge scoring 1–5 against each of the 42 Inferact cases — from Continuous Batching And Pagedattention Kv Cache Behavior (22 scenarios) down to the smallest pack — its own expected behavior, with critical-severity flags and negative controls. Only judge-passed evals are published to the Inferact library.

How many test cases does the Inferact library include?+

The Inferact eval library includes 42 graded test cases across 3 eval packs, the largest being Continuous Batching And Pagedattention Kv Cache Behavior with 22 scenarios. Each case defines an input, expected behavior, and pass/fail criteria.

How do I run these evals against Inferact or my own agent?+

Request the library with your work email above and we'll set it up in a Corsac workspace, where you can run all 3 Inferact packs — Continuous Batching And Pagedattention Kv Cache Behavior and Openai Compatible Api Conformance and the rest — against Inferact or your own agent with your own data.