Eval directory

Evals for EXO

Eval coverage for EXO, mapped from its public product surface.

About EXO

exo is software that links Macs and workstations into a single local inference cluster for running frontier AI models without the cloud. It discovers devices automatically, reads network topology, and splits models across the combined memory of multiple machines. It exposes standard OpenAI-, Claude-, Responses-, and Ollama-compatible APIs, and is distributed under Apache-2.0 for macOS 26+.

Industry

local AI inference clustering software

Website

exolabs.net

Use the eval library for EXO

We'll build out the full library — runnable test cases with inputs, expected behavior, and pass/fail checks — in your Corsac workspace.

All evals →

More AI eval libraries

Coverage map

What would you measure for EXO?

6 scoring areas · 24 capabilities mapped · grounded in 8 cited pages

Every eval set is graded on

Adversarial robustness
Workflow quality
Safety gates
Operator quality

Pass/Fail + LLM judge 1–5 · critical severity flags · negative controls

Device discovery and cluster formation

How machines running exo find each other and become a usable cluster without hand-built configuration.

“Machines running exo discover each other without hand-built cluster config.” exolabs.net

Mapped capabilities

4 capabilities

Automatic peer discovery
Explaining that exo instances discover each other rather than requiring manual cluster config files.
Cluster membership changes
Behavior and user expectations when a machine joins or leaves an already-running cluster.
Setup expectations for a first cluster
What a user must actually do to go from two installed machines to one addressable cluster.
Discovery troubleshooting
Guidance when a machine is running exo but does not appear as part of the cluster.

Topology and interconnect awareness

How exo reads the network it is running on and what it reports about the links between devices.

“It finds devices, reads the network, splits models across memory, and gives you normal APIs.” exolabs.net

Mapped capabilities

4 capabilities

Tracked topology attributes
Memory, link type, latency, bandwidth, and available compute as the properties exo tracks per device.
Link type differences
How different interconnects between machines change what the cluster can reasonably do.
RDMA over Thunderbolt 5
Accurate representation of the published 8 microsecond latency figure and the setup it describes.
Heterogeneous device mixes
Reasoning about clusters whose machines differ in memory and compute.

Model placement across memory

Splitting a model across the combined memory of multiple devices instead of fitting it on one box.

“Models can be split across devices instead of fitting on one box.” exolabs.net

Mapped capabilities

4 capabilities

Sharding across combined memory
Explaining that models can be split across devices rather than requiring a single machine to hold them.
Fit and capacity reasoning
Helping a user reason about whether a target model can run on the hardware they have.
Placement decisions and topology
How the device and link properties exo reads relate to where shards are placed.
Single-machine versus cluster tradeoffs
When splitting across machines is the point and when one box is already sufficient.

API compatibility surface

Serving OpenAI-, Claude-, Responses-, and Ollama-compatible APIs to normal clients.

“macOS 26+ · Apache-2.0 · OpenAI, Claude, Responses, and Ollama compatible” exolabs.net

Mapped capabilities

4 capabilities

OpenAI-compatible endpoint
Pointing an existing OpenAI SDK or tool at a local exo endpoint.
Claude and Responses compatibility
Which client shapes are covered by the published compatibility list.
Ollama-compatible clients
Using tools built against the Ollama API against an exo cluster.
Compatibility boundaries
Not overclaiming support for client shapes outside the four published API surfaces.

Illustrative example

Input: I have a Python app using the OpenAI SDK. Can I keep the code and just run it against my exo cluster instead of the cloud?
Expected behavior: Confirms exo serves an OpenAI-compatible API, so the client keeps working by pointing its base URL at the local cluster instead of the cloud, and notes Claude, Responses, and Ollama compatible clients are also supported. No cloud account is needed.

Local-first operation and distribution terms

Running frontier models without a cloud account, under the published platform and license terms.

Mapped capabilities

4 capabilities

No cloud account required
Accurately describing that inference runs on the user's own hardware.
macOS 26+ requirement
Stating the published platform floor and declining to promise unlisted platforms.
Apache-2.0 licensing
Answering license and redistribution questions against the published license.
Data locality expectations
What staying local does and does not imply for a user's prompts and files.

Illustrative example

Input: We have Macs plus a few Windows and Linux workstations. Can we put them all in one exo cluster, and what license is exo under?
Expected behavior: States that exo is published for macOS 26 or later and does not promise official support for Windows or Linux, so those machines fall outside the documented platform. Confirms exo is distributed under Apache-2.0.

Models, artifacts, and community channels

Where models and compression artifacts come from and where users go for help.

Mapped capabilities

4 capabilities

Hugging Face model sources
Distinguishing the EXO Labs org releases from the separate 0xSero artifacts.
Quantized and REAP artifacts
Representing compression artifacts and their published download counts without embellishment.
Support routing
Directing setup questions and hardware results to Discord, and release news to X.
Published metrics accuracy
Repeating star, fork, model, and download counts only as published snapshots.

Coverage is mapped from EXO's public pages (8 crawled). Examples are illustrative, not real test cases. The runnable eval library — graded inputs, expected behavior, and pass/fail checks — is built when you request it above.

Frequently asked questions

What do the Corsac evals for EXO test?+

The coverage map is generated from EXO's own public product surface (local AI inference clustering software): 6 scoring areas — Device discovery and cluster formation, Topology and interconnect awareness, and Model placement across memory, and more — spanning 24 mapped capabilities, each graded on adversarial robustness, workflow quality, safety gates, and operator quality once the library is built.

How are the EXO evals scored?+

Every case generated for EXO — across Device discovery and cluster formation and Topology and interconnect awareness and the other mapped areas — is graded with pass/fail checks plus an LLM judge scoring 1–5 against its expected behavior, with critical-severity flags and negative controls. Only judge-passed evals are published.

How many test cases does the EXO library include?+

The full EXO library is built on request. The coverage map spans 6 areas and 24 capabilities (for example, Automatic peer discovery and Cluster membership changes under Device discovery and cluster formation); each becomes graded test cases — inputs, expected behavior, pass/fail checks — in your Corsac workspace.

How do I run these evals against EXO or my own agent?+

Request the library with your work email above. We'll build out all 6 mapped EXO areas and set them up in a Corsac workspace, where you can run every test case against EXO or your own agent with your own data.