Eval directory

Evals for OpenClaw

Eval coverage for OpenClaw, mapped from its public product surface.

About OpenClaw

OpenClaw is an open-source personal AI assistant that runs locally on a user's own machine and works with the apps they already use. It installs via a one-line script or npm across macOS, Linux, and Windows, with beta companion apps for macOS and Windows providing gateway, chat, setup, and node controls. It extends through skills (reusable agent procedures reviewed via Skill Workshop) distributed on ClawHub, and is now stewarded by the OpenClaw Foundation with extended-stable release channels and a public maturity scorecard.

Industry

open-source personal AI assistant / agent runtime

Website

openclaw.ai

Use the eval library for OpenClaw

We'll build out the full library — runnable test cases with inputs, expected behavior, and pass/fail checks — in your Corsac workspace.

All evals →

More AI eval libraries

Coverage map

What would you measure for OpenClaw?

6 scoring areas · 23 capabilities mapped · grounded in 8 cited pages

Every eval set is graded on

Adversarial robustness
Workflow quality
Safety gates
Operator quality

Pass/Fail + LLM judge 1–5 · critical severity flags · negative controls

Installation & Onboarding

Getting OpenClaw onto a machine across macOS, Linux, and Windows via the one-line installer, npm, or a source checkout, and completing first-run onboarding.

“Works on macOS, Linux, and Windows.” openclaw.ai

Mapped capabilities

4 capabilities

One-liner vs npm install paths
curl installer, npm i -g openclaw, and what the one-liner provisions (including Node.js) for the user
Source checkout install
git clone with corepack/pnpm workspace setup and running via pnpm openclaw
First-run onboarding
openclaw onboard flow and the macOS Administrator/Homebrew prompt on first run
Platform prerequisites
stated OS support: macOS, Linux, Windows, plus companion app minimums (macOS 15+, Windows 10 20H2+/11)

Release Channels & Maturity

Choosing and persisting an update channel, and interpreting the maturity scorecard when deciding what to rely on.

“Extended-stable releases will roll out once a month.” openclaw.ai

Mapped capabilities

4 capabilities

Channel switching
openclaw update --channel dev | stable | extended-stable and which commands persist the channel
Extended-stable semantics
monthly cadence, YYYY.M.33 starting version, patch increments for backported fixes, minimum one-month support window
Maturity scorecard lookup
features organized by surface area and category, and where a given feature sits in the maturity model
Upgrade guidance for critical workloads
advising extended-stable vs stable vs dev based on stability needs

Illustrative example

Input: I run OpenClaw on a box doing real work and want backported security fixes without surprise changes. How do I move to extended-stable and keep it there?
Expected behavior: The response gives `openclaw update --channel extended-stable` as the way to persist the channel, and notes that `npm install -g openclaw@extended-stable` installs that release but does not persist the update channel.

Skill Workshop (Authoring Loop)

The proposal-first review step that sits in front of any change to reusable agent procedures.

“While it is pending, the file is PROPOSAL.md, not SKILL.md.” openclaw.ai

Mapped capabilities

4 capabilities

Proposal creation
turning a repeated request into a drafted, inactive proposal rather than a hand-written file
Proposal vs active skill state
PROPOSAL.md while pending, SKILL.md once applied, and the agent not running it in the interim
Revise and iterate
user feedback rounds that update the pending draft before it takes effect
Apply or reject
cleanly activating or discarding a proposal, including its support files

Illustrative example

Input: The agent just drafted a skill from my weekly inbox routine. Is it running yet, and what file is it stored as?
Expected behavior: The response states the draft is a pending proposal that is not active and will not run yet, stored as PROPOSAL.md, and that it becomes SKILL.md only after the user applies it.

ClawHub Distribution & Skill Provenance

Discovering and vetting skills distributed through ClawHub, including the documentation and scanning that ship with them.

“Every ClawHub skill now ships with a Skill Card documenting what the skill does” openclaw.ai

Mapped capabilities

3 capabilities

Skill Card contents
what the skill does and where it came from, shipped with every ClawHub skill
SkillSpector scanning
scanning for hidden instructions and other agentic risks before a skill is trusted
Provenance-based trust decisions
explaining to a user what is and is not verified about a third-party skill

Exec Approvals & Host Guardrails

Opt-in controls over what the agent is allowed to execute on the host, and how humans stay in the decision path.

Mapped capabilities

4 capabilities

Auto mode opt-in
enabling auto mode for host exec approvals and its opt-in, non-default nature
Policy-first evaluation
policy runs before execution; low-risk misses are routed to review
Human-in-the-loop review
preserving human approval for actions policy does not clear
Remote action approvals
approving agent actions from the paired iOS/Android app rather than at the host

Gateway, Nodes & Companion Clients

Operating the gateway and node features from native companion apps and mobile clients paired to a user's own gateway.

Mapped capabilities

4 capabilities

Companion app controls
gateway, chat, setup, and node controls in the beta macOS app and Windows Hub tray
Pairing mobile to your own gateway
iOS and Android apps with chat and Talk mode paired to a self-hosted gateway
Node mode operation
running and controlling node features from the desktop clients
Windows containment
Windows node and gateway running inside Microsoft Execution Containers

Coverage is mapped from OpenClaw's public pages (8 crawled). Examples are illustrative, not real test cases. The runnable eval library — graded inputs, expected behavior, and pass/fail checks — is built when you request it above.

Frequently asked questions

What do the Corsac evals for OpenClaw test?+

The coverage map is generated from OpenClaw's own public product surface (open-source personal AI assistant / agent runtime): 6 scoring areas — Installation & Onboarding, Release Channels & Maturity, and Skill Workshop (Authoring Loop), and more — spanning 23 mapped capabilities, each graded on adversarial robustness, workflow quality, safety gates, and operator quality once the library is built.

How are the OpenClaw evals scored?+

Every case generated for OpenClaw — across Installation & Onboarding and Release Channels & Maturity and the other mapped areas — is graded with pass/fail checks plus an LLM judge scoring 1–5 against its expected behavior, with critical-severity flags and negative controls. Only judge-passed evals are published.

How many test cases does the OpenClaw library include?+

The full OpenClaw library is built on request. The coverage map spans 6 areas and 23 capabilities (for example, One-liner vs npm install paths and Source checkout install under Installation & Onboarding); each becomes graded test cases — inputs, expected behavior, pass/fail checks — in your Corsac workspace.

How do I run these evals against OpenClaw or my own agent?+

Request the library with your work email above. We'll build out all 6 mapped OpenClaw areas and set them up in a Corsac workspace, where you can run every test case against OpenClaw or your own agent with your own data.