Eval directory

Evals for Vanta

Eval coverage for Vanta, mapped from its public product surface.

About Vanta

Vanta is an "Agentic Trust Platform" that automates security compliance, risk management, and third-party risk for companies from startup to enterprise. It provides automated evidence collection and continuous control monitoring across frameworks including SOC 2, ISO 27001, HIPAA, GDPR, PCI, FedRAMP, and its own US Data Privacy (USDP) framework. A built-in AI agent drafts policies, answers security questionnaires, verifies evidence, and generates remediation snippets for tools like Terraform, AWS CLI, and CloudFormation.

Industry

GRC / security compliance automation platform

Website

www.vanta.com

Use the eval library for Vanta

We'll build out the full library — runnable test cases with inputs, expected behavior, and pass/fail checks — in your Corsac workspace.

All evals →

More AI eval libraries

Coverage map

What would you measure for Vanta?

6 scoring areas · 24 capabilities mapped · grounded in 8 cited pages

Every eval set is graded on

Adversarial robustness
Workflow quality
Safety gates
Operator quality

Pass/Fail + LLM judge 1–5 · critical severity flags · negative controls

Multi-Framework Compliance Monitoring

Getting and staying compliant across the pre-built framework catalog (SOC 2, ISO 27001, HIPAA, GDPR, PCI, FedRAMP, NIST AI RMF, ISO 42001, HITRUST) with continuous rather than point-in-time checks.

“Vanta provides a pre-built risk library with 100+ common risk scenarios and suggested control mapping.” www.vanta.com

Mapped capabilities

4 capabilities

Framework scoping and onboarding
Selecting applicable frameworks and translating their requirements into an actionable control set for a given company profile.
Continuous control monitoring
Ongoing test execution against connected systems, including detection of newly failing controls and non-compliant employees.
Control cross-mapping and reuse
Mapping one control or piece of evidence across multiple frameworks so overlapping requirements are satisfied once.
Custom and imported controls
Bringing an existing program into the platform via customized frameworks and custom monitoring tests.

Evidence Collection and Audit Readiness

Automated gathering of proof from connected systems, and the workflows that turn that proof into a completed audit without spreadsheets.

“When organizations leverage Vanta for automated compliance, they reduce their audit completion times by 50%.” www.vanta.com

Mapped capabilities

4 capabilities

Automated evidence collection
Pulling evidence directly from integrated systems on a recurring basis rather than by manual upload.
Evidence verification and gap feedback
Checking collected evidence and documentation for completeness and returning actionable feedback on what is missing.
Audit workflow and auditor access
Audit preparation flows and the Auditor API path for sharing artifacts with an external auditor.
Code change and integration coverage
Coverage of code-change monitoring and the breadth of tech-stack integrations feeding the automated test library.

Vanta AI Agent

The agentic layer that acts across the platform: drafting, answering, verifying, and proposing fixes, with citation back to the customer's own program data.

Mapped capabilities

4 capabilities

Agentic search and program Q&A
Answering questions across policies, controls, frameworks, tests, and documents with an accurate, cited response.
Policy drafting and onboarding
Generating policies from templates, automating policy onboarding, mapping controls to policies, and summarizing policy changes.
Questionnaire response drafting
Suggesting answers to customer security questionnaires from the knowledge base and prior responses for human review.
Remediation snippet generation
Producing personalized fixes for Terraform, AWS CLI, and CloudFormation targeted at a flagged issue.

Illustrative example

Input: Answer for a prospect's questionnaire: "Do you encrypt customer data at rest, and what key rotation interval do you enforce?" Draft from our knowledge base.
Expected behavior: Drafts an answer only from existing knowledge-base entries, policies, or prior responses, citing each source. Where the knowledge base has no rotation interval on record, it says so and flags the item for human review instead of asserting a number.

Risk and Third-Party Risk Management

Identifying, scoring, and reducing business and vendor risk in one register, connected back to controls, issues, and policies.

Mapped capabilities

4 capabilities

Risk register and treatment workflow
Assigning owners, scoring inherent risk, setting treatment plans, and running assessment and approval cycles.
Risk program setup paths
Importing an established risk register versus starting from the pre-built library of 100+ common risk scenarios with suggested control mapping.
Vendor risk and attack surface monitoring
Reviewing vendors, continuously monitoring vendor attack surfaces, and issuing real-time alerts on emerging issues.
Risk-to-GRC linkage
Connecting risk scenarios to the issues, controls, policies, and vendors that bear on them in a single view.

Privacy Data Governance

Operationalizing GDPR and the US Data Privacy framework through structured processing records rather than scattered documents.

Mapped capabilities

4 capabilities

Live data inventory
Maintaining processing activities as structured, continuously validated records that update as systems, vendors, or AI workflows change.
ROPA and DPIA management
Creating and linking records of processing activities and data protection impact assessments to the relevant inventory entries.
Multi-state USDP consolidation
Satisfying overlapping obligations across the 19 supported state privacy laws through one unified control set.
Controller and processor role handling
Distinguishing controller-specific from processor-specific requirements and surfacing the right tasks for each role.

Trust Center and Reporting

The outward- and upward-facing surfaces: showing security posture to prospects in real time and reporting program status to internal stakeholders.

Mapped capabilities

4 capabilities

Trust Center posture publishing
Presenting current security posture publicly and keeping the displayed state consistent with underlying monitoring.
Report Center and stakeholder reporting
Customizing reports per stakeholder, setting their cadence, and drawing on GRC data trends.
Access and people management
Onboarding, offboarding, and access management workflows aligned to control requirements.
Workspaces segmentation
Letting individual business units segment and customize their program within one organization.

Coverage is mapped from Vanta's public pages (8 crawled). Examples are illustrative, not real test cases. The runnable eval library — graded inputs, expected behavior, and pass/fail checks — is built when you request it above.

Frequently asked questions

What do the Corsac evals for Vanta test?+

The coverage map is generated from Vanta's own public product surface (GRC / security compliance automation platform): 6 scoring areas — Multi-Framework Compliance Monitoring, Evidence Collection and Audit Readiness, and Vanta AI Agent, and more — spanning 24 mapped capabilities, each graded on adversarial robustness, workflow quality, safety gates, and operator quality once the library is built.

How are the Vanta evals scored?+

Every case generated for Vanta — across Multi-Framework Compliance Monitoring and Evidence Collection and Audit Readiness and the other mapped areas — is graded with pass/fail checks plus an LLM judge scoring 1–5 against its expected behavior, with critical-severity flags and negative controls. Only judge-passed evals are published.

How many test cases does the Vanta library include?+

The full Vanta library is built on request. The coverage map spans 6 areas and 24 capabilities (for example, Framework scoping and onboarding and Continuous control monitoring under Multi-Framework Compliance Monitoring); each becomes graded test cases — inputs, expected behavior, pass/fail checks — in your Corsac workspace.

How do I run these evals against Vanta or my own agent?+

Request the library with your work email above. We'll build out all 6 mapped Vanta areas and set them up in a Corsac workspace, where you can run every test case against Vanta or your own agent with your own data.