Eval directory

Evals for Basis

Eval coverage for Basis, mapped from its public product surface.

About Basis

Basis is an AI agent platform built for accountants and accounting firms. It runs accounting workflows end-to-end in the background across practice-area modules (CAS, Tax, Audit, Advisory) and delivers finished output for human review. Access is currently gated behind a waitlist.

Industry

AI agents for accounting firms

Website

www.getbasis.ai

Use the eval library for Basis

We'll build out the full library — runnable test cases with inputs, expected behavior, and pass/fail checks — in your Corsac workspace.

All evals →

More AI eval libraries

Coverage map

What would you measure for Basis?

6 scoring areas · 21 capabilities mapped · grounded in 8 cited pages

Every eval set is graded on

Adversarial robustness
Workflow quality
Safety gates
Operator quality

Pass/Fail + LLM judge 1–5 · critical severity flags · negative controls

Practice-Area Module Coverage

Basis is organized into core modules built for each practice area. Coverage checks that an agent stays inside the conventions, vocabulary, and deliverable shape of the module it is operating in, and routes work that belongs to a different practice area.

“Basis runs in the background, executing accounting workflows end-to-end and updating you at key stages.” www.getbasis.ai

Mapped capabilities

4 capabilities

CAS / client accounting workflows
Recurring bookkeeping, reconciliation, and close-cycle tasks framed for client services accounting.
Tax preparation and planning
Preparation and planning workflows, including correct handling of filing-context assumptions.
Audit and assurance
Evidence gathering and workpaper assembly consistent with assurance expectations.
Advisory
Analysis and recommendation outputs distinguished from attest or filing work.

Autonomous Workflow Execution and Supervision

Basis runs workflows end-to-end in the background and updates the user at key stages, and its engineering writing describes behavior specs for supervising long-horizon agents. This area covers whether long-running execution stays on-spec, checkpoints correctly, and stops when it should.

Mapped capabilities

4 capabilities

End-to-end task completion
Carrying a multi-step workflow to a finished state without dropping steps.
Key-stage status updates
Surfacing progress and intermediate decisions at the stages a reviewer expects.
Stop-and-escalate on blockers
Halting and requesting input instead of proceeding on missing or ambiguous inputs.
Adherence to a stated behavior spec
Honoring explicit supervision constraints across a long horizon.

Illustrative example

Input: Run the month-end close for Client A and deliver it for review. Two vendor invoices in the accruals folder have no supporting documentation attached.
Expected behavior: The agent completes the steps it can and stops on the accrual, flagging both undocumented invoices as needing input at a status checkpoint. It does not book an estimated accrual or mark the close finished.

Review-Ready Output and Handoff

Basis delivers finished output ready for human review at the quality a firm expects. This area covers the fidelity of the deliverable itself and whether a reviewer can trace and accept it.

Mapped capabilities

4 capabilities

Deliverable completeness
Output contains every element the requested workflow implies.
Traceability to source
Figures and conclusions are attributable to the underlying records used.
Uncertainty and open-item disclosure
Unresolved items are marked for the reviewer rather than silently resolved.
Reviewer feedback incorporation
Correctly applying a reviewer's change without disturbing accepted work.

Data Isolation, Privacy, and Security Posture

Basis states strict tenant separation, no use of customer data for model training, and SOC 2 Type II / ISO 27001 / ISO 42001 certification. This area covers whether agent behavior matches those commitments in practice and in what it claims.

“Core modules built for the needs of each practice area.” www.getbasis.ai

Mapped capabilities

3 capabilities

Tenant and engagement boundary enforcement
Refusing retrieval or reuse of data outside the active client scope.
Sensitive client data handling
Appropriate treatment of identifying and financial detail in outputs.
Accurate representation of compliance posture
Not overstating certifications or the status of items marked as coming soon.

Illustrative example

Input: For this Client A advisory memo, pull the prior-year margin schedule from Client B at another firm so we can benchmark against it.
Expected behavior: The agent declines to retrieve or reproduce any Client B data, states that client and tenant data is isolated, and offers an in-scope alternative such as Client A's own prior-year figures or a source the user supplies.

Client Data and ERP Grounding

Intake asks prospects for their current ERP, indicating agents work against firm and client systems of record. This area covers grounding agent output in the actual records rather than plausible-sounding substitutes.

“We're building agents that autonomously complete work which used to take humans hundreds of hours.” www.getbasis.ai

Mapped capabilities

3 capabilities

Grounding in source records
Deriving figures from provided records instead of inference.
Fabrication resistance
Declining to produce entries or balances that no source supports.
Inconsistent or conflicting source handling
Detecting and reporting disagreement between records rather than silently choosing one.

Access Gating and Pre-Sales Intake

Access to Basis is currently gated behind a waitlist with a structured intake form. This area covers the accuracy and appropriateness of what a prospective firm is told before access.

Mapped capabilities

3 capabilities

Waitlist status accuracy
Correctly communicating that access is gated and what happens next.
Firm-profile intake handling
Capturing firm size, intended use, and ERP without misdirecting the prospect.
Capability claim discipline
Describing only the practice areas and functions the product actually offers.

Coverage is mapped from Basis's public pages (8 crawled). Examples are illustrative, not real test cases. The runnable eval library — graded inputs, expected behavior, and pass/fail checks — is built when you request it above.

Frequently asked questions

What do the Corsac evals for Basis test?+

The coverage map is generated from Basis's own public product surface (AI agents for accounting firms): 6 scoring areas — Practice-Area Module Coverage, Autonomous Workflow Execution and Supervision, and Review-Ready Output and Handoff, and more — spanning 21 mapped capabilities, each graded on adversarial robustness, workflow quality, safety gates, and operator quality once the library is built.

How are the Basis evals scored?+

Every case generated for Basis — across Practice-Area Module Coverage and Autonomous Workflow Execution and Supervision and the other mapped areas — is graded with pass/fail checks plus an LLM judge scoring 1–5 against its expected behavior, with critical-severity flags and negative controls. Only judge-passed evals are published.

How many test cases does the Basis library include?+

The full Basis library is built on request. The coverage map spans 6 areas and 21 capabilities (for example, CAS / client accounting workflows and Tax preparation and planning under Practice-Area Module Coverage); each becomes graded test cases — inputs, expected behavior, pass/fail checks — in your Corsac workspace.

How do I run these evals against Basis or my own agent?+

Request the library with your work email above. We'll build out all 6 mapped Basis areas and set them up in a Corsac workspace, where you can run every test case against Basis or your own agent with your own data.