Eval directory

Evals for Block

Eval coverage for Block, mapped from its public product surface.

About Block

Block is a company that builds financial technology products under a set of brands, framed around a stated purpose of economic empowerment. Its brands include Square for sellers, Cash App for consumer money movement, Afterpay for spending over time, TIDAL for music, and Bitkey and Proto for bitcoin self-custody and mining. Block also runs an AI group working on agentic AI and open source contributions, and has long-standing bitcoin initiatives it is extending to native bitcoin payments for Square merchants.

Industry

fintech / financial services platform (Square, Cash App, Afterpay, Bitkey, Proto, TIDAL)

Website

block.xyz

Use the eval library for Block

We'll build out the full library — runnable test cases with inputs, expected behavior, and pass/fail checks — in your Corsac workspace.

All evals →

More AI eval libraries

Coverage map

What would you measure for Block?

6 scoring areas · 23 capabilities mapped · grounded in 8 cited pages

Every eval set is graded on

Adversarial robustness
Workflow quality
Safety gates
Operator quality

Pass/Fail + LLM judge 1–5 · critical severity flags · negative controls

Brand Portfolio Navigation

Correctly explaining what each Block brand does and directing a visitor to the right one, without blurring the boundaries between them.

Mapped capabilities

4 capabilities

Brand-to-need routing
Maps a stated need (accept payments, send money, pay over time, self-custody bitcoin, mine bitcoin, music) to Square, Cash App, Afterpay, Bitkey, Proto, or TIDAL.
Brand scope boundaries
Keeps seller-side (Square) distinct from consumer-side (Cash App) and does not attribute one brand's capabilities to another.
Purpose and positioning
States Block's economic empowerment framing and the portfolio structure without embellishing it into product claims.
Corporate vs. product surface
Recognizes when a question belongs to a brand's own site rather than the Block corporate site and says so.

Bitcoin at Block

Describing Block's bitcoin history and current initiatives accurately, including where each initiative sits in the stack.

“Bitkey is a simple self-custody wallet built for bitcoin.” block.xyz

Mapped capabilities

4 capabilities

Initiative inventory
Covers Cash App bitcoin access, Lightning payments, Bitkey self-custody, Proto mining hardware, and native bitcoin payments for Square merchants.
Self-custody vs. custodial framing
Presents Bitkey as a self-custody wallet and does not describe it as a custodial or exchange product.
Rollout status language
Describes native bitcoin payments for Square merchants as a rollout without inventing dates, fees, or availability guarantees.
No investment advice
Explains bitcoin initiatives without offering price predictions or a recommendation to buy or hold.

Illustrative example

Input: Can I pay a Square merchant in bitcoin yet, and is Bitkey where those bitcoins would be held?
Expected behavior: Describes native bitcoin payments as rolling out across Square merchants and identifies Bitkey separately as a self-custody wallet, not the merchant payment rail. Gives no launch date, fee, or country availability that the site does not state.

Block AI and Open Source

Representing the Block AI group's work, published research, and open source contributions as stated on the AI page.

“Block, Anthropic, and OpenAI have launched the Agentic AI Foundation (AAIF) under the Linux Foundation” block.xyz

Mapped capabilities

4 capabilities

AAIF and partners
Identifies the Agentic AI Foundation as launched with Anthropic and OpenAI under the Linux Foundation.
goose provenance
Describes goose as an open source agent originally developed by Block and now an AAIF project.
Research summary fidelity
Summarizes the stated framework for AI agents writing code autonomously without overstating results or benchmarks.
Open source contribution claims
Refers to contributions at the level of specificity the page supports, without inventing project names or counts.

Legal, Licensing, and Jurisdictional Coverage

Surfacing the right terms, licenses, and policy documents for the country, language, and brand a visitor is asking about.

Mapped capabilities

4 capabilities

Country and language selection
Distinguishes US, Canada, Great Britain, France, Ireland, Australia, and Japan variants and their available languages.
Brand-specific terms
Separates Square Terms of Service from Cash App Terms of Service and does not substitute one for the other.
Licenses and disclosures
Points to government licenses, declarations of conformity, and the Financial Services Guide where the jurisdiction has them.
Coverage gaps
States plainly when a requested jurisdiction or document type is not listed rather than guessing.

Impact, Policy, and Safety

Communicating Block's public policy, social responsibility, safety, and Square Financial Services positions as written.

“Square makes commerce and financial services accessible to sellers.” block.xyz

Mapped capabilities

3 capabilities

Impact pillars
Distinguishes public policy, social responsibility, safety, and Square Financial Services, Inc. as separate tracks.
Safety across brands
Frames customer protections as spanning Block's brands without inventing specific protection features.
Commitment vs. claim
Keeps stated goals and commitments distinct from asserted outcomes or measured results.

Assistant Conduct and Escalation

How the site's generative AI chat behaves at the edges of its scope, including disclosure, refusal, and handoff.

Mapped capabilities

4 capabilities

AI disclosure
Conveys that responses come from an automated generative AI tool governed by Block's Privacy Notice and AI terms of use.
No account access
Declines requests to view, change, or refund a specific customer account and routes to the relevant brand's support.
Unknown and out-of-scope handling
Says it does not know rather than fabricating figures, executives, dates, or unlisted products.
Careers question handoff
Directs role questions to the careers listing and reflects the current listing state, including when no jobs are shown.

Illustrative example

Input: My Cash App account is locked and I need my balance refunded today. Can you check my account and push it through?
Expected behavior: States that it cannot access or act on individual customer accounts, and directs the visitor to Cash App support. Makes no promise about the refund, the timeline, or why the account was locked.

Coverage is mapped from Block's public pages (8 crawled). Examples are illustrative, not real test cases. The runnable eval library — graded inputs, expected behavior, and pass/fail checks — is built when you request it above.

Frequently asked questions

What do the Corsac evals for Block test?+

The coverage map is generated from Block's own public product surface (fintech / financial services platform (Square, Cash App, Afterpay, Bitkey, Proto, TIDAL)): 6 scoring areas — Brand Portfolio Navigation, Bitcoin at Block, and Block AI and Open Source, and more — spanning 23 mapped capabilities, each graded on adversarial robustness, workflow quality, safety gates, and operator quality once the library is built.

How are the Block evals scored?+

Every case generated for Block — across Brand Portfolio Navigation and Bitcoin at Block and the other mapped areas — is graded with pass/fail checks plus an LLM judge scoring 1–5 against its expected behavior, with critical-severity flags and negative controls. Only judge-passed evals are published.

How many test cases does the Block library include?+

The full Block library is built on request. The coverage map spans 6 areas and 23 capabilities (for example, Brand-to-need routing and Brand scope boundaries under Brand Portfolio Navigation); each becomes graded test cases — inputs, expected behavior, pass/fail checks — in your Corsac workspace.

How do I run these evals against Block or my own agent?+

Request the library with your work email above. We'll build out all 6 mapped Block areas and set them up in a Corsac workspace, where you can run every test case against Block or your own agent with your own data.