Eval directory

Evals for Endor Labs

Eval coverage for Endor Labs, mapped from its public product surface.

About Endor Labs

Endor Labs is an application security platform built around securing AI coding agents and the code they produce. Its AURI capability helps agents write secure code and governs the actions they take, spanning AI SAST code scanning, open source/SCA with reachability, secrets detection, malware detection, CI/CD pipeline discovery, and SBOM management. It is sold in Developer (free), Core, and Pro tiers with per-product selection.

Industry

agentic application security (AppSec) platform

Website

www.endorlabs.com

Use the eval library for Endor Labs

We'll build out the full library — runnable test cases with inputs, expected behavior, and pass/fail checks — in your Corsac workspace.

All evals →

More AI eval libraries

Coverage map

What would you measure for Endor Labs?

6 scoring areas · 22 capabilities mapped · grounded in 8 cited pages

Every eval set is graded on

Adversarial robustness
Workflow quality
Safety gates
Operator quality

Pass/Fail + LLM judge 1–5 · critical severity flags · negative controls

AURI agentic coding security

The capability that helps coding agents write secure code from the start and governs the actions those agents take, addressing the blind spots the product names: privileged agent access, insecure patterns learned from open source, and unsafe commands.

“AURI helps agents write secure code from the start and governs the actions they take, without slowing builders down.” www.endorlabs.com

Mapped capabilities

4 capabilities

Secure-by-default code assistance
AURI steering an agent toward secure patterns as code is written, rather than only flagging after the fact.
Agent action governance
Governing the actions an agent takes when it holds privileged access to a repository or environment.
Unsafe command and data-exposure guardrails
Preventing agents from exposing sensitive data or credentials, or running unsafe commands.
Developer-velocity preservation
Applying governance without slowing builders down, per the product's stated positioning.

First-party code scanning (AI SAST)

Consolidated scanning of code the organization owns, including AI SAST, business logic and design review, and agentic remediation as described in the Code product bundle.

“scan and fix vulnerabilities and exposed secrets in your code as you work—no account required” www.endorlabs.com

Mapped capabilities

4 capabilities

AI SAST finding generation
Detecting vulnerabilities in first-party code.
Business logic and design review
Review that goes beyond pattern matching to logic and design concerns.
Agentic remediation
Proposing and applying fixes for detected issues.
Fix-as-you-work loop
The free-tier flow where an agent scans and fixes vulnerabilities in code with no account required.

Open source risk and reachability

Securing dependencies and container images, with reachability analysis used to reduce noise and prioritize what actually matters, plus malware detection on open source packages.

“Secure dependencies (SCA) and container images with reachability, malware detection” www.endorlabs.com

Mapped capabilities

4 capabilities

Dependency (SCA) findings
Identifying vulnerable open source dependencies.
Reachability-based prioritization
Using reachability to distinguish exploitable findings from noise.
Container image coverage
Extending open source risk analysis to container images.
Malware detection in packages
Flagging malicious packages, including fast-spreading npm compromises of the kind the research team publishes on.

Illustrative example

Input: Our scan flagged a critical CVE in a transitive dependency. Nothing in our code calls the affected function. Should we treat this as urgent?
Expected behavior: Explains that reachability analysis is what determines priority here: a vulnerable function that is never reached is deprioritized rather than treated as urgent, while a reachable path raises it. Notes reachability is a Core-tier capability.

Secrets detection

Detecting exposed secrets and credentials in code, offered in the Code product bundle and available in the free Developer tier alongside vulnerability scanning.

“Consolidate code scanning (AI SAST), business logic & design reviews, secrets detection, and agentic remediation.” www.endorlabs.com

Mapped capabilities

3 capabilities

Exposed credential detection
Finding secrets committed to or generated in code.
Agent-triggered secret exposure
Secrets surfaced in the context of coding agents with privileged access.
In-editor remediation flow
Fixing exposed secrets as part of the developer's working loop.

CI/CD pipeline discovery and coverage

Automatic discovery of everything that touches code from SCM through build, test, and deploy, producing a map of security tool coverage and gaps, and treating CI workflows such as GitHub Actions as dependencies.

“automatically discovers everything that touches your code, from your source code management system” www.endorlabs.com

Mapped capabilities

4 capabilities

Pipeline and tool discovery
Discovering CI/CD, SCA, SAST, container scan, and IaC tooling in use across the organization.
Coverage gap mapping
Showing where insecure gaps exist in pipeline security coverage.
Shadow engineering visibility
Surfacing unofficial repositories and unsanctioned or misconfigured tools and marketplace extensions.
GitHub Actions risk assessment
Treating pre-built, often open source Actions as dependencies subject to risk analysis.

Illustrative example

Input: We use about forty pre-built GitHub Actions across our pipelines. Does Endor Labs cover those, or only the code in our repos?
Expected behavior: Confirms CI/CD pipeline discovery covers what touches code from SCM through build, test, and deploy, and that pre-built Actions are analyzed as dependencies rather than trusted implicitly. Mentions the resulting coverage map of tools and gaps.

SBOM management and packaging

SBOM Hub's import, standardization, and continuous monitoring of SBOMs across CycloneDX and SPDX, together with the tiered Developer/Core/Pro model and per-product selection that determines what a given customer can access.

“With support for a wide range of versions for the two most popular formats - CycloneDX and SPDX.” www.endorlabs.com

Mapped capabilities

3 capabilities

SBOM ingestion and format support
Manual upload and CI-automated ingestion across a range of CycloneDX and SPDX versions.
Continuous SBOM monitoring
Keeping component license, security, and operational risk current as new advisories and CVEs land.
Tier and product-selection boundaries
What Developer (free), Core, and Pro each unlock, and how per-product selection across Code and Open Source applies.

Coverage is mapped from Endor Labs's public pages (8 crawled). Examples are illustrative, not real test cases. The runnable eval library — graded inputs, expected behavior, and pass/fail checks — is built when you request it above.

Frequently asked questions

What do the Corsac evals for Endor Labs test?+

The coverage map is generated from Endor Labs's own public product surface (agentic application security (AppSec) platform): 6 scoring areas — AURI agentic coding security, First-party code scanning (AI SAST), and Open source risk and reachability, and more — spanning 22 mapped capabilities, each graded on adversarial robustness, workflow quality, safety gates, and operator quality once the library is built.

How are the Endor Labs evals scored?+

Every case generated for Endor Labs — across AURI agentic coding security and First-party code scanning (AI SAST) and the other mapped areas — is graded with pass/fail checks plus an LLM judge scoring 1–5 against its expected behavior, with critical-severity flags and negative controls. Only judge-passed evals are published.

How many test cases does the Endor Labs library include?+

The full Endor Labs library is built on request. The coverage map spans 6 areas and 22 capabilities (for example, Secure-by-default code assistance and Agent action governance under AURI agentic coding security); each becomes graded test cases — inputs, expected behavior, pass/fail checks — in your Corsac workspace.

How do I run these evals against Endor Labs or my own agent?+

Request the library with your work email above. We'll build out all 6 mapped Endor Labs areas and set them up in a Corsac workspace, where you can run every test case against Endor Labs or your own agent with your own data.