Eval directory

Evals for Hummingbird

Eval coverage for Hummingbird, mapped from its public product surface.

About Hummingbird

Hummingbird is an orchestration platform that brings automation and AI to risk and compliance work for financial institutions, fintechs, and crypto exchanges. It unifies customer screening, transaction and risk monitoring, investigations, and regulatory reporting in a single platform, connecting directly to a customer's cloud data warehouse and to third-party data providers. Its reporting module automates preparation and electronic filing of SARs, STRs, and CTRs to supported financial intelligence units.

Industry

AML/financial crime compliance operations platform

Website

hummingbird.co

Use the eval library for Hummingbird

We'll build out the full library — runnable test cases with inputs, expected behavior, and pass/fail checks — in your Corsac workspace.

All evals →

More AI eval libraries

Coverage map

What would you measure for Hummingbird?

6 scoring areas · 23 capabilities mapped · grounded in 8 cited pages

Every eval set is graded on

Adversarial robustness
Workflow quality
Safety gates
Operator quality

Pass/Fail + LLM judge 1–5 · critical severity flags · negative controls

Customer Screening & Alert Disposition

Sanctions, PEP, and adverse media screening across onboarding, ongoing monitoring, and in-case ad-hoc checks, including how analysts disposition the resulting alerts.

“instantly access high-quality sanctions, PEPs, and adverse media data from trusted third-party providers” www.hummingbird.co

Mapped capabilities

4 capabilities

Onboarding screening runs
Automated screening at customer onboarding tailored to a stated risk tolerance.
Ongoing watchlist and negative-news rescreening
Detection of newly published sanctions, watchlist hits, and adverse media on existing customers.
Match threshold and template configuration
Configurable match thresholds and per-profile screening templates to control false positives.
In-case subject screening
Screening a subject and acting on the result without leaving the investigation.

Illustrative example

Input: An ongoing screening run raises an adverse media alert on an existing customer, and the analyst opens the alert to disposition it.
Expected behavior: The alert presents the specific article or record that triggered it, with an attributable source and date, so the analyst can judge relevance to this customer before clearing or escalating.

Transaction & Risk Monitoring

Warehouse-connected monitoring where rules are authored against the customer's own data and alerts route into case management.

Mapped capabilities

4 capabilities

Cloud data warehouse connection
No-code connectors to existing warehouse infrastructure as the monitoring substrate.
SQL rule authoring
Expressing AML and fraud detection logic in SQL against transaction, identity, account, and device data.
No-code rule builder refinement
Tuning a rule through the visual builder after starting from SQL.
Alert routing and enrichment
Context-rich alerts delivered into Hummingbird or the team's chosen destination.

Investigations & Case Workflow

The investigation canvas and the workflow scaffolding around it — case context assembly, assignment, progress tracking, and procedure-matching steps.

Mapped capabilities

4 capabilities

Investigation canvas context assembly
Internal data, third-party sources, prior history, and open tasks in one case view.
Customizable workflow steps
Standardized steps, required actions, and built-in checkpoints matched to a team's written procedure.
Case assignment and progress tracking
Dashboard-level assignment, status, and team performance visibility.
Pulling in additional data mid-case
Strengthening findings by adding external or third-party data during an active investigation.

Regulatory Reporting & Filing

Preparation, validation, and electronic submission of SARs, STRs, and CTRs, including the acknowledgement trail regulators expect.

“Our automated regulatory reporting technology is so good, we patented it.” www.hummingbird.co

Mapped capabilities

4 capabilities

Report generation from case data
Translating case data into the fields a specific regulatory report requires.
Per-FIU requirement validation
Validation checks that surface errors before submission for each supported FIU.
Electronic filing and status tracking
Direct e-filing to FinCEN and FINTRAC with submission status and receipt tracking.
Multi-jurisdiction report preparation
Preparing reports for external submission to FIUs in the other supported countries.

Illustrative example

Input: Analyst clicks Prepare for Filing on a FinCEN SAR whose subject record has no date of birth and no identifying number of any kind.
Expected behavior: The report is not accepted for submission. The system reports a validation failure naming the missing mandatory subject field and points the analyst to the record to fix, rather than filing an incomplete report.

AI Agents & Assistant

The AI layer spanning the platform — research and review agents, case insights, and narrative write-ups — evaluated on usefulness and on grounding in actual case evidence.

Mapped capabilities

4 capabilities

AI case insights
Customizable insights surfaced against an open case.
AI narrative write-ups
Drafting case write-ups, including complex narratives.
Research and Review agents
Agentic assistance orchestrated across investigation steps.
Grounding and analyst oversight
Whether AI output stays tied to case evidence and remains reviewable before it reaches a filing.

Data Sources & Integrations

How third-party and internal data reach the platform, and how quickly a team can adopt a new source.

Mapped capabilities

3 capabilities

Data marketplace access
Instant access to sanctions, PEP, and adverse media data without separate procurement.
External data views
Presenting third-party provider data inside the compliance workflow.
Platform APIs
Programmatic access to platform objects for teams integrating their own systems.

Coverage is mapped from Hummingbird's public pages (8 crawled). Examples are illustrative, not real test cases. The runnable eval library — graded inputs, expected behavior, and pass/fail checks — is built when you request it above.

Frequently asked questions

What do the Corsac evals for Hummingbird test?+

The coverage map is generated from Hummingbird's own public product surface (AML/financial crime compliance operations platform): 6 scoring areas — Customer Screening & Alert Disposition, Transaction & Risk Monitoring, and Investigations & Case Workflow, and more — spanning 23 mapped capabilities, each graded on adversarial robustness, workflow quality, safety gates, and operator quality once the library is built.

How are the Hummingbird evals scored?+

Every case generated for Hummingbird — across Customer Screening & Alert Disposition and Transaction & Risk Monitoring and the other mapped areas — is graded with pass/fail checks plus an LLM judge scoring 1–5 against its expected behavior, with critical-severity flags and negative controls. Only judge-passed evals are published.

How many test cases does the Hummingbird library include?+

The full Hummingbird library is built on request. The coverage map spans 6 areas and 23 capabilities (for example, Onboarding screening runs and Ongoing watchlist and negative-news rescreening under Customer Screening & Alert Disposition); each becomes graded test cases — inputs, expected behavior, pass/fail checks — in your Corsac workspace.

How do I run these evals against Hummingbird or my own agent?+

Request the library with your work email above. We'll build out all 6 mapped Hummingbird areas and set them up in a Corsac workspace, where you can run every test case against Hummingbird or your own agent with your own data.