Eval directory

Evals for LithosAI

Eval coverage for LithosAI, mapped from its public product surface.

About LithosAI

LithosAI is an inference engine that serves large open-weight coding models at very high throughput for agentic harnesses like Claude Code and Codex. The site advertises 1,000 tokens per second per user serving Kimi K2.7 Code (1T parameters) on 8×B200 GPUs. It positions inference speed, rather than model quality alone, as the differentiator for agents running in a loop.

Industry

high-speed LLM inference for coding agents

Website

www.lithosai.com

Use the eval library for LithosAI

We'll build out the full library — runnable test cases with inputs, expected behavior, and pass/fail checks — in your Corsac workspace.

All evals →

More AI eval libraries

Coverage map

What would you measure for LithosAI?

6 scoring areas · 24 capabilities mapped · grounded in 8 cited pages

Every eval set is graded on

Adversarial robustness
Workflow quality
Safety gates
Operator quality

Pass/Fail + LLM judge 1–5 · critical severity flags · negative controls

Per-User Throughput and Latency

The headline claim: 1,000 output tokens per second per user serving a 1T-parameter model, versus 174-291 tok/s from major providers. Covers sustained rate, not just peak.

“1,000 tokens per second” www.lithosai.com

Mapped capabilities

4 capabilities

Sustained per-user output token rate
Steady-state tokens/second measured for a single user session on Kimi K2.7 Code, not aggregate cluster throughput.
Time to first token
Prefill latency for typical agentic prompts, which the site's tok/s framing does not itself cover.
Throughput stability under concurrency
Whether the per-user rate holds as additional concurrent sessions share the 8xB200 deployment.
Long-context throughput behavior
How output rate changes as the conversation context grows over a long agentic session.

Illustrative example

Input: A single client streams one completion request to Kimi K2.7 Code on the Lithos endpoint and generates at least 2,000 output tokens with no other load on the deployment.
Expected behavior: The endpoint streams continuously to completion and sustains roughly 1,000 output tokens per second for that single user, consistent with the rate advertised on the landing page.

Model Fidelity Under Acceleration

The site claims model-native precision, zero approximation, and full model quality. This area checks that the speedup is not paid for with quantization or output degradation.

“Model-native precision. Zero approximation. Full model quality.” www.lithosai.com

Mapped capabilities

4 capabilities

Output equivalence versus reference deployment
Deterministic-decoding comparison of Lithos-served outputs against a full-precision reference of the same model.
Numerical precision of serving path
Evidence that no quantization or approximate attention/decoding is applied relative to model-native precision.
Coding task quality retention
Code-generation quality on the served model held against the same model's expected baseline.
Determinism and reproducibility
Repeatability of outputs at fixed sampling settings across repeated identical requests.

Illustrative example

Input: Send the same coding prompt at temperature 0 with a fixed seed to the Lithos endpoint and to a model-native full-precision deployment of Kimi K2.7 Code.
Expected behavior: Both deployments return the same completion, since Lithos states it applies zero approximation and preserves full model quality rather than trading precision for speed.

Harness Compatibility

LithosAI positions itself behind Claude Code, Codex, and 'your harness'. This area covers whether it drops into those harnesses as an inference backend without behavioral surprises.

“Claude Code • Codex • Your harness” www.lithosai.com

Mapped capabilities

4 capabilities

Claude Code integration
Configuring Claude Code to route inference to Lithos and completing an end-to-end task.
Codex integration
Same drop-in path for Codex as an alternate agentic harness.
Custom harness / API surface conformance
Whether the endpoint honors the request and streaming contract a third-party harness expects.
Tool-call and structured-output handling
Fidelity of tool-call formatting that agentic harnesses depend on to drive their loops.

Agentic Loop Behavior

The product's stated thesis is that inference runs inside a loop, so speed decides how fast real work gets done. This area covers behavior across many sequential turns rather than single calls.

Mapped capabilities

4 capabilities

Multi-turn session performance
Whether throughput and quality hold across dozens of sequential agent turns in one task.
End-to-end task wall-clock time
Total time for a realistic agentic coding task, the metric the speed claim implies.
Streaming continuity
Uninterrupted token streaming to a harness that consumes output incrementally.
Long-running session stability
Behavior over extended agent runs, including context accumulation and session lifetime.

Failure Handling and Recovery

An inference backend inserted into an agent loop becomes a dependency for every turn. This area covers what a harness observes when a request cannot be served normally.

Mapped capabilities

4 capabilities

Mid-stream interruption behavior
What the harness receives when a streaming response is cut off partway.
Capacity and rate-limit signaling
Clarity of the signal returned when the deployment cannot admit a request.
Error surface legibility to harnesses
Whether failures arrive in a form an agentic harness can distinguish and retry on.
Retry and resumption semantics
Whether a retried request is safe and produces coherent continuation of the agent's work.

Performance Claim Substantiation and Access

The public surface is a marketing site plus a blog making specific comparative numbers, with a GET ACCESS call to action as the only user workflow.

Mapped capabilities

4 capabilities

Reproducibility of the 3.4-5.7x comparison
Whether the stated baseline of 174-291 tok/s and the resulting multiple are defined well enough to reproduce.
Disclosure of benchmark conditions
Hardware, model version, context length, and concurrency assumptions behind the published figures.
Access request workflow
The GET ACCESS path from the landing page through to a usable endpoint.
Blog and documentation accuracy
Consistency between claims in the published posts and the landing page figures.

Coverage is mapped from LithosAI's public pages (8 crawled). Examples are illustrative, not real test cases. The runnable eval library — graded inputs, expected behavior, and pass/fail checks — is built when you request it above.

Frequently asked questions

What do the Corsac evals for LithosAI test?+

The coverage map is generated from LithosAI's own public product surface (high-speed LLM inference for coding agents): 6 scoring areas — Per-User Throughput and Latency, Model Fidelity Under Acceleration, and Harness Compatibility, and more — spanning 24 mapped capabilities, each graded on adversarial robustness, workflow quality, safety gates, and operator quality once the library is built.

How are the LithosAI evals scored?+

Every case generated for LithosAI — across Per-User Throughput and Latency and Model Fidelity Under Acceleration and the other mapped areas — is graded with pass/fail checks plus an LLM judge scoring 1–5 against its expected behavior, with critical-severity flags and negative controls. Only judge-passed evals are published.

How many test cases does the LithosAI library include?+

The full LithosAI library is built on request. The coverage map spans 6 areas and 24 capabilities (for example, Sustained per-user output token rate and Time to first token under Per-User Throughput and Latency); each becomes graded test cases — inputs, expected behavior, pass/fail checks — in your Corsac workspace.

How do I run these evals against LithosAI or my own agent?+

Request the library with your work email above. We'll build out all 6 mapped LithosAI areas and set them up in a Corsac workspace, where you can run every test case against LithosAI or your own agent with your own data.