Eval directory

Evals for Arcee AI

Eval coverage for Arcee AI, mapped from its public product surface.

About Arcee AI

Trinity is an open-weight model family from Arcee AI, an American AI research lab, spanning Trinity Nano, Mini, Large, and Large-Thinking for on-device through frontier reasoning use. Models can be consumed via Arcee's API or downloaded and self-hosted, with weights released for inspection and fine-tuning under the OpenMDW-1.1 license. Arcee also builds derived models such as Genesis-Science-1, an open scientific research model developed with the U.S.

Employees

30 across the whole company (14 on research)

Industry

open-weight foundation models / model API

Website

www.arcee.ai

Use the eval library for Arcee AI

We'll build out the full library — runnable test cases with inputs, expected behavior, and pass/fail checks — in your Corsac workspace.

All evals →

More AI eval libraries

Coverage map

What would you measure for Arcee AI?

6 scoring areas · 24 capabilities mapped · grounded in 8 cited pages

Every eval set is graded on

Adversarial robustness
Workflow quality
Safety gates
Operator quality

Pass/Fail + LLM judge 1–5 · critical severity flags · negative controls

Model family and tier selection

Correctly distinguishing the four Trinity tiers and matching each to a workload, from ultra-lightweight on-device use through frontier reasoning, without inventing tiers or capabilities the material does not claim.

“Ultra-lightweight and fast enough to run on-device.” www.arcee.ai

Mapped capabilities

4 capabilities

Tier-to-workload mapping
Nano for on-device/lightweight, Mini as the balanced everyday workhorse, Large and Large-Thinking for heavier and frontier reasoning tasks.
Reasoning-tier positioning
When Large-Thinking's frontier reasoning is warranted versus a cheaper tier, per its stated positioning for the hardest problems and long-running agents.
Quantized variant awareness
Recognizing that quantized variants ship alongside each Trinity release and where they fit in a deployment decision.
Family boundary discipline
Not fabricating tier names, sizes, or benchmark numbers beyond what Arcee publishes for the Trinity family.

Illustrative example

Input: I need one model that runs offline on a laptop and a second for long multi-step reasoning. Which Trinity models should I use, and how do I get them?
Expected behavior: Recommends Trinity Nano for the on-device offline case and Trinity-Large-Thinking for the long reasoning workload, and notes both can be downloaded as open weights and self-hosted rather than requiring the hosted API.

Deployment paths: hosted API vs. self-hosting

Guiding a user between shipping on Arcee's API and downloading the weights to run themselves, including what each path requires and what it does not lock the user into.

Mapped capabilities

4 capabilities

API onboarding path
Getting an API key and building against the hosted endpoint as the fastest route to production.
Self-hosted download path
Downloading open weights and running them on the user's own hardware, including local machines, VPS, and dedicated servers.
Provider selection in downstream tools
Choosing an inference provider or model backend when a client tool supports multiple options.
Path trade-off explanation
Explaining the trade-offs between hosted and self-hosted without claiming exclusivity or unsupported SLAs.

Open-weight licensing and compliance

Answering OpenMDW-1.1 questions accurately: the single Linux Foundation license now covering the entire Trinity family and all artifacts in a model distribution, and what teams may do under it.

“Every model is open-weight, so you can inspect, fine-tune, and go to production on your terms.” www.arcee.ai

Mapped capabilities

4 capabilities

License identification
Naming OpenMDW-1.1 as the current license for Nano, Mini, Large, Large-Thinking, and quantized variants, including retroactively updated releases.
Artifact coverage
Explaining that the license spans weights, architecture, configs, documentation, code, and evaluation materials as one instrument rather than artifact by artifact.
Permitted downstream use
Confirming inspection, fine-tuning, redistribution, adaptation, and production use for developers and enterprises.
Comparison to code-only licenses
Explaining why Apache 2.0 and similar source-code licenses leave ambiguity across model artifacts, without misstating Trinity's actual license.

Illustrative example

Input: Can I redistribute a quantized Trinity Mini checkpoint I fine-tuned, and what license covers the config and evaluation files shipped alongside the weights?
Expected behavior: Names OpenMDW-1.1 as the single license covering the whole Trinity distribution, including quantized variants, configs, and evaluation materials, and confirms fine-tuning and redistribution are permitted. It should not state that Apache 2.0 governs Trinity's weights.

Agentic operation with Hermes

Supporting the documented Hermes agent workflow powered by Trinity-Large-Thinking, from installation through configuration of backends, tools, and messaging surfaces.

Mapped capabilities

4 capabilities

Installation and setup wizard
The single-command install, the virtual environment it creates, and the five-step setup roadmap.
Model and backend configuration
Selecting model and provider and configuring the terminal backend and agent settings.
Tool configuration
Enabling tools such as web search and image generation within the agent.
Messaging platform linking
Optionally connecting Telegram or Discord to interact with the agent remotely.

Tool use and structured, auditable output

Behaviors exercised in the Loka/Arcee/AWS post-training work: seeking evidence with tools, reasoning across incomplete evidence, and returning output another researcher can check.

Mapped capabilities

4 capabilities

Evidence-seeking tool calls
Invoking biomedical and research tools to gather evidence rather than answering from memory alone.
Strict JSON conformance
Returning schema-valid structured responses, as in the Gene Ontology annotation task where JSON validity is scored.
Reasoning under incomplete evidence
Producing defensible inferences when the retrieved evidence is partial, without overclaiming.
Checkable citations and identifiers
Emitting citations, identifiers, and tool traces that a reviewer can independently verify.

Distribution, provenance, and program claims

Accurately representing where Trinity artifacts live and the status of adjacent efforts, so users are not sent to the wrong place or told an unreleased model is available.

Mapped capabilities

4 capabilities

Hugging Face as the artifact home
The Hub as the exclusive home for Arcee's models, datasets, and agent traces.
Genesis-Science-1 status
GS1 as a DOE-partnered, trillion-parameter-class open model built on next-generation Trinity, announced as releasing openly later this year with weights and a technical report.
Builders and partner programs
Pointing to the Trinity Builders Program and partner touchpoints such as contacting sales when a request exceeds self-serve.
Provenance claim discipline
Not asserting availability, benchmark results, or partnerships beyond what Arcee has published.

Coverage is mapped from Arcee AI's public pages (8 crawled). Examples are illustrative, not real test cases. The runnable eval library — graded inputs, expected behavior, and pass/fail checks — is built when you request it above.

Frequently asked questions

What do the Corsac evals for Arcee AI test?+

The coverage map is generated from Arcee AI's own public product surface (open-weight foundation models / model API): 6 scoring areas — Model family and tier selection, Deployment paths: hosted API vs. self-hosting, and Open-weight licensing and compliance, and more — spanning 24 mapped capabilities, each graded on adversarial robustness, workflow quality, safety gates, and operator quality once the library is built.

How are the Arcee AI evals scored?+

Every case generated for Arcee AI — across Model family and tier selection and Deployment paths: hosted API vs. self-hosting and the other mapped areas — is graded with pass/fail checks plus an LLM judge scoring 1–5 against its expected behavior, with critical-severity flags and negative controls. Only judge-passed evals are published.

How many test cases does the Arcee AI library include?+

The full Arcee AI library is built on request. The coverage map spans 6 areas and 24 capabilities (for example, Tier-to-workload mapping and Reasoning-tier positioning under Model family and tier selection); each becomes graded test cases — inputs, expected behavior, pass/fail checks — in your Corsac workspace.

How do I run these evals against Arcee AI or my own agent?+

Request the library with your work email above. We'll build out all 6 mapped Arcee AI areas and set them up in a Corsac workspace, where you can run every test case against Arcee AI or your own agent with your own data.