Eval directory

Evals for ZeroPath

Q: What do the Corsac evals for ZeroPath test?

The coverage map is generated from ZeroPath's own public product surface (AI-native application security (SAST/SCA) platform): 6 scoring areas — AI-Native SAST & Code Analysis, Dependency & Supply Chain (SCA), and Infrastructure as Code & Cloud Policy, and more — spanning 24 mapped capabilities, each graded on adversarial robustness, workflow quality, safety gates, and operator quality once the library is built.

Q: How are the ZeroPath evals scored?

Every case generated for ZeroPath — across AI-Native SAST & Code Analysis and Dependency & Supply Chain (SCA) and the other mapped areas — is graded with pass/fail checks plus an LLM judge scoring 1–5 against its expected behavior, with critical-severity flags and negative controls. Only judge-passed evals are published.

Q: How many test cases does the ZeroPath library include?

The full ZeroPath library is built on request. The coverage map spans 6 areas and 24 capabilities (for example, Business logic and authorization flaws and False-positive discipline under AI-Native SAST & Code Analysis); each becomes graded test cases — inputs, expected behavior, pass/fail checks — in your Corsac workspace.

Q: How do I run these evals against ZeroPath or my own agent?

Request the library with your work email above. We'll build out all 6 mapped ZeroPath areas and set them up in a Corsac workspace, where you can run every test case against ZeroPath or your own agent with your own data.

Eval coverage for ZeroPath, mapped from its public product surface.

About ZeroPath

ZeroPath is an AI-native application security platform that scans code, dependencies, infrastructure-as-code, and secrets across GitHub, GitLab, Bitbucket, and Azure DevOps. It uses LLM-driven analysis to find business logic and authentication flaws, ranks findings by reachability and real-world exploit intelligence (CISA KEV, FIRST EPSS), and opens ready-to-merge fix PRs. An agent called Zero sits on top of that scanning to route findings to owners, propose fixes, and track remediation against SLAs.

Industry

AI-native application security (SAST/SCA) platform

Website

zeropath.com

Use the eval library for ZeroPath

We'll build out the full library — runnable test cases with inputs, expected behavior, and pass/fail checks — in your Corsac workspace.

All evals →

More AI eval libraries

Coverage map

What would you measure for ZeroPath?

6 scoring areas · 24 capabilities mapped · grounded in 8 cited pages

Every eval set is graded on

Adversarial robustness
Workflow quality
Safety gates
Operator quality

Pass/Fail + LLM judge 1–5 · critical severity flags · negative controls

AI-Native SAST & Code Analysis

LLM-driven detection of vulnerabilities in first-party code, with emphasis on business logic and authentication flaws that rule-based scanners miss, and on keeping false positives low across 15+ languages and unfamiliar frameworks.

“Find critical vulnerabilities that legacy tools miss. 75% fewer false positives.” zeropath.com

Mapped capabilities

4 capabilities

Business logic and authorization flaws
IDOR, authorization bypass paths, missing authentication checks, and race conditions in payment flows.
False-positive discipline
Whether flagged code is genuinely exploitable, and whether non-issues are dismissed with sound reasoning rather than hedged.
Framework-agnostic detection
Behavior on frameworks and vulnerability classes not covered by a rule database, per the LLM-driven claim.
Finding explanation quality
Natural-language vulnerability reports with a traceable path from source to sink.

Illustrative example

Input: A REST handler loads a record by an ID taken from the request path and returns it after checking only that the caller is authenticated.
Expected behavior: The finding names the issue as an authorization flaw (IDOR/broken object-level authorization), traces the untrusted ID from the request to the data access call, and recommends an ownership or tenancy check rather than input sanitization.

Dependency & Supply Chain (SCA)

Inventory across 35+ package ecosystems, plus the exploitability layer that separates a present CVE from one reachable in your code and under active exploitation in the wild.

Mapped capabilities

4 capabilities

Reachability and usage-based risk
Whether the vulnerable code path described by the CVE is actually invoked by the application.
Exploit intelligence enrichment
CISA KEV known-exploited status with listing date and ransomware-campaign flag; FIRST EPSS probability and percentile.
Ecosystem coverage
npm/yarn/pnpm, PyPI, Maven/Gradle, Go modules, Cargo, NuGet, Composer, Docker/OCI.
Upgrade PR correctness
Whether the proposed dependency bump resolves the CVE without breaking dependents.

Illustrative example

Input: A repo pins a library with a CVE that the application imports but never calls on the vulnerable path. The CVE appears on CISA KEV.
Expected behavior: The finding reports reachability and exploit-in-the-wild status as separate signals: the vulnerable path is not reachable, while the KEV listing is still surfaced with its listing date. Priority reflects the unreachable path rather than defaulting to critical on KEV alone.

Infrastructure as Code & Cloud Policy

Pre-deploy scanning of infrastructure definitions against 500+ built-in policies plus organization-specific rules, mapped to compliance frameworks.

“Comprehensive security scanning for your infrastructure code with 500+ built-in policies.” zeropath.com

Mapped capabilities

4 capabilities

Misconfiguration detection
Exposed databases, unencrypted storage, and excessive IAM permissions across AWS, Azure, GCP, and Kubernetes.
Multi-format parsing
Terraform, CloudFormation, ARM templates, Kubernetes YAML, Helm charts, Dockerfiles, and Kustomize.
Custom Rego (OPA) policies
Author, evaluate, and enforce organization-specific policies alongside built-ins.
Compliance mapping
CIS Benchmarks, PCI-DSS, HIPAA, SOC 2, NIST, and GDPR checks tied to concrete resources.

Zero: Agentic Remediation Workflow

The persistent agent layer that takes a confirmed finding through ownership resolution, ticketing, fix proposal, and SLA escalation across teams, repos, and tools.

Mapped capabilities

4 capabilities

Ownership resolution and routing
Identify the owning team and open tickets in Linear, Jira, or Slack without manual triage.
Fix PRs and detection rules
Ready-to-merge remediation PRs matching project style, plus rules that catch the same pattern org-wide.
SLA tracking and escalation
Monitor remediation progress and escalate before deadlines are missed.
External report validation
Validate an inbound bug bounty report against source code and confirm or refute it.

Risk Prioritization, Analytics & Reporting

Turning raw findings into a ranked queue and into metrics leadership can act on, with attribution back to the change that introduced each issue.

Mapped capabilities

4 capabilities

Distinct prioritization signals
AI-assessed CVSS 4.0, KEV listing, EPSS probability, and reachability reported separately with transparent reasoning.
Attribution to introducing change
Line-level git blame and PR-author tracking linking a vulnerability to its origin.
Metrics dashboards
MTTR, common vulnerability classes, issues by language, and team performance.
Stakeholder exports
PDF and DOCX reports of security metrics and findings for compliance audiences.

Platform Integration, Deployment & Access Control

How the platform meets an org where it runs: SCM and CI/CD wiring, PR-time scan latency, identity controls, and deployment models including self-hosted and BYOK.

“<60s Average PR scan time” zeropath.com

Mapped capabilities

4 capabilities

SCM and CI/CD integration
GitHub, GitLab, Bitbucket, and Azure DevOps Services, with PR reviews in the pipeline.
PR scan latency
Sub-60-second average PR scan behavior on incremental diffs.
Identity and provisioning
SSO/SAML and SCIM provisioning for enterprise tenants.
Deployment and key models
On-prem, self-hosted, and private cloud, with bring-your-own LLM keys.

Coverage is mapped from ZeroPath's public pages (8 crawled). Examples are illustrative, not real test cases. The runnable eval library — graded inputs, expected behavior, and pass/fail checks — is built when you request it above.

Frequently asked questions

What do the Corsac evals for ZeroPath test?+

The coverage map is generated from ZeroPath's own public product surface (AI-native application security (SAST/SCA) platform): 6 scoring areas — AI-Native SAST & Code Analysis, Dependency & Supply Chain (SCA), and Infrastructure as Code & Cloud Policy, and more — spanning 24 mapped capabilities, each graded on adversarial robustness, workflow quality, safety gates, and operator quality once the library is built.

How are the ZeroPath evals scored?+

Every case generated for ZeroPath — across AI-Native SAST & Code Analysis and Dependency & Supply Chain (SCA) and the other mapped areas — is graded with pass/fail checks plus an LLM judge scoring 1–5 against its expected behavior, with critical-severity flags and negative controls. Only judge-passed evals are published.

How many test cases does the ZeroPath library include?+

The full ZeroPath library is built on request. The coverage map spans 6 areas and 24 capabilities (for example, Business logic and authorization flaws and False-positive discipline under AI-Native SAST & Code Analysis); each becomes graded test cases — inputs, expected behavior, pass/fail checks — in your Corsac workspace.

How do I run these evals against ZeroPath or my own agent?+

Request the library with your work email above. We'll build out all 6 mapped ZeroPath areas and set them up in a Corsac workspace, where you can run every test case against ZeroPath or your own agent with your own data.