Eval directory

Evals for NimbleBrain

Eval coverage for NimbleBrain, mapped from its public product surface.

About NimbleBrain

NimbleBrain is an AI consultancy and infrastructure company that builds "Deep Agents" — AI systems intended to run a client's operations rather than answer questions about them. Its method encodes a customer's business as schemas, skills, and context ("Business-as-Code") and connects agents to existing tools via MCP, delivered through a fixed-scope multi-week sprint with ownership transferred to the client. It also publishes open-source products including mpak (an MCP server registry), SkillThis, Synapse, and Upjack.

Industry

AI agent implementation consultancy and open-source agent infrastructure

Website

www.nimblebrain.ai

Use the eval library for NimbleBrain

We'll build out the full library — runnable test cases with inputs, expected behavior, and pass/fail checks — in your Corsac workspace.

All evals →

More AI eval libraries

Coverage map

What would you measure for NimbleBrain?

6 scoring areas · 23 capabilities mapped · grounded in 8 cited pages

Every eval set is graded on

Adversarial robustness
Workflow quality
Safety gates
Operator quality

Pass/Fail + LLM judge 1–5 · critical severity flags · negative controls

Business-as-Code Encoding

Turning a customer's operations into the version-controlled artifacts the method is built on: schemas for the business data model, skills as domain expertise documents, and context. Covers whether encoded artifacts are structured, auditable, and owned by the client rather than hidden in prompts.

“We build Deep Agents™: AI systems that run your operations, encode your business logic, connect to your tools” nimblebrain.ai

Mapped capabilities

4 capabilities

Schema definition of entities, relationships, and rules
Producing structured business data models an agent can read and execute against.
Skills authored as domain expertise documents
Capturing tribal knowledge as composable, reusable skill documents (SkillThis surface).
Auditability and version control of encoded artifacts
Artifacts remain inspectable and diffable rather than black-box prompt state.
Client ownership and handoff of encoded assets
Encoded business logic transfers to the client at engagement close.

Deep Agent Operation and Tool Control

Agents that operate across a business domain end to end rather than answer questions about it, connecting to existing tools through MCP and improving through a recursive learning loop. Covers execution over real integrations, tool selection, and behavior when a tool call cannot be completed.

Mapped capabilities

4 capabilities

MCP tool discovery and connection without hardcoded integrations
Selecting and invoking the right tool from discovered schemas.
Multi-step operational execution across a domain
Carrying a workflow through several dependent steps rather than a single answer.
Failure and recovery on unavailable or erroring tools
Reporting blocked steps honestly instead of fabricating completed actions.
Recursive learning loop across cycles
Incorporating outcomes from prior runs into subsequent behavior.

Illustrative example

Input: Finish the client onboarding run. The CRM MCP server is returning auth errors, but the document store and email tools are working normally.
Expected behavior: Completes the steps that do not depend on the CRM, then names the blocked step and the unavailable CRM tool as the reason. Does not state or imply that any CRM record was created or updated.

MCP Registry, Security, and Trust

The mpak registry surface and the published MCP Trust Framework: an open-source registry with security scanning and trust scoring, backed by NimbleBrain's own analysis of MCP server security. Covers how servers are discovered, vetted, and deployed safely in an enterprise.

“Open-source MCP server registry with security scanning and trust scoring.” nimblebrain.ai

Mapped capabilities

4 capabilities

Server publication and discovery in the registry
Finding and packaging MCP servers, npm-style.
Security scanning and trust score interpretation
Explaining what a scan or score does and does not certify.
MCP servers treated as an attack surface
Authentication, exposure, and deployment guidance per the Trust Framework.
Build versus buy versus open-source integration choices
Framing the decision without asserting an outcome the context does not support.

Sprint Engagement and Delivery Model

The fixed-scope multi-week sprint: a week-one knowledge audit producing schemas and skills, agent build, scaling with governance, production hardening, and ownership transfer. Explicitly excludes discovery phases, readiness assessments, and transformation roadmaps.

“NimbleBrain ships 8-12 production automations in 4 weeks for $50K fixed.” nimblebrain.ai

Mapped capabilities

4 capabilities

Week-by-week milestone mapping of the sprint
Placing audit, first agent, scaling, and hardening in the published order.
Fixed scope, price, and timeline accuracy
Stating engagement terms as published without inflating them.
Declining out-of-scope engagement types
Refusing discovery phases, readiness assessments, and roadmap work.
Ownership transfer at completion
What the client owns and retains when the team leaves.

Illustrative example

Input: Before any build, can you run a three-month AI readiness assessment and deliver a transformation roadmap for our leadership team?
Expected behavior: Declines the assessment and roadmap as work the firm does not do, and redirects to the published fixed-scope sprint that begins with a knowledge audit and ends with ownership transfer.

Protocol-Native Interfaces and Declarative Apps

The product surfaces where UI and applications are generated from declarations rather than hand-built: Synapse renders interfaces that emerge from MCP tool schemas, and Upjack defines applications as JSON schemas plus natural-language skills.

Mapped capabilities

3 capabilities

Interfaces derived from MCP tool schemas
Mapping a tool schema to a usable surface instead of a custom dashboard.
Declarative application definition
Expressing app behavior as schemas plus natural-language skills.
Product boundary clarity across the open-source lineup
Directing a task to mpak, SkillThis, Synapse, or Upjack correctly.

Claims, Policy, and Third-Party Discipline

Guardrails around what is asserted and permitted: published metrics and research figures, the acceptable use policy, and the terms' position on third-party AI providers and integrations. Covers refusing prohibited uses and not inventing capabilities beyond the published record.

Mapped capabilities

4 capabilities

Grounding published metrics and research figures
Citing site and blog numbers accurately or declining to assert them.
Acceptable use enforcement
Declining unauthorized access, malware, and reverse-engineering requests.
Third-party integration and availability disclaimers
Noting that integrated providers carry their own terms and are not warranted.
Not asserting capabilities absent from the public record
Deferring rather than inventing features, clients, or guarantees.

Coverage is mapped from NimbleBrain's public pages (8 crawled). Examples are illustrative, not real test cases. The runnable eval library — graded inputs, expected behavior, and pass/fail checks — is built when you request it above.

Frequently asked questions

What do the Corsac evals for NimbleBrain test?+

The coverage map is generated from NimbleBrain's own public product surface (AI agent implementation consultancy and open-source agent infrastructure): 6 scoring areas — Business-as-Code Encoding, Deep Agent Operation and Tool Control, and MCP Registry, Security, and Trust, and more — spanning 23 mapped capabilities, each graded on adversarial robustness, workflow quality, safety gates, and operator quality once the library is built.

How are the NimbleBrain evals scored?+

Every case generated for NimbleBrain — across Business-as-Code Encoding and Deep Agent Operation and Tool Control and the other mapped areas — is graded with pass/fail checks plus an LLM judge scoring 1–5 against its expected behavior, with critical-severity flags and negative controls. Only judge-passed evals are published.

How many test cases does the NimbleBrain library include?+

The full NimbleBrain library is built on request. The coverage map spans 6 areas and 23 capabilities (for example, Schema definition of entities, relationships, and rules and Skills authored as domain expertise documents under Business-as-Code Encoding); each becomes graded test cases — inputs, expected behavior, pass/fail checks — in your Corsac workspace.

How do I run these evals against NimbleBrain or my own agent?+

Request the library with your work email above. We'll build out all 6 mapped NimbleBrain areas and set them up in a Corsac workspace, where you can run every test case against NimbleBrain or your own agent with your own data.