Eval directory

Evals for Semgrep

Eval coverage for Semgrep, mapped from its public product surface.

About Semgrep

Semgrep is an application security platform that scans source code, open source dependencies, and secrets for vulnerabilities. Its product line includes Semgrep Code (SAST), Semgrep Supply Chain (SCA and malware blocking), Semgrep Secrets, and Semgrep Guardian for AI-generated code, unified in the Semgrep AppSec Platform. The pages also promote AI-assisted triage and remediation, workflow automation, broad CI/developer-tool integrations, and a migration program for teams moving off Checkmarx or Snyk.

Industry

application security (AppSec) platform — SAST, SCA, and secrets detection

Website

semgrep.dev

Use the eval library for Semgrep

We'll build out the full library — runnable test cases with inputs, expected behavior, and pass/fail checks — in your Corsac workspace.

All evals →

More AI eval libraries

Coverage map

What would you measure for Semgrep?

6 scoring areas · 22 capabilities mapped · grounded in 8 cited pages

Every eval set is graded on

Adversarial robustness
Workflow quality
Safety gates
Operator quality

Pass/Fail + LLM judge 1–5 · critical severity flags · negative controls

Semgrep Code (SAST)

Static application security testing over first-party source code, including the language breadth claimed for Pro Engine and the positioning around surfacing 'the issues that matter' rather than raw finding volume.

“Find and fix the issues that matter in your code (SAST)” semgrep.dev

Mapped capabilities

4 capabilities

Source-code vulnerability detection
Explaining what Semgrep Code scans and the classes of code issues it is positioned to find.
Pro Engine language coverage
Answering which engine tier is associated with support across 30+ enterprise languages.
OWASP Top 10 positioning
Connecting SAST scanning to the advertised OWASP Top 10 critical web risk coverage.
Finding prioritization framing
Describing the 'find and fix the issues that matter' claim without overstating accuracy guarantees.

Supply Chain & Malware Protection

Scanning of open source dependencies for vulnerabilities plus blocking of malicious packages, including the open-source malware protection and fintech supply-chain risk solution framings.

“Fix vulnerabilities in open source dependencies and block malware” semgrep.dev

Mapped capabilities

3 capabilities

Open source dependency vulnerabilities
Identifying Semgrep Supply Chain as the SCA product and what it inspects.
Malware blocking
Distinguishing malicious-package blocking from ordinary CVE reporting.
Supply chain attack solution framing
Mapping the software supply chain attack and fintech risk solution pages to the right product.

Illustrative example

Input: We want to stop a malicious npm package from reaching our build, and separately we keep finding AWS keys committed in code. Which Semgrep products cover those two things?
Expected behavior: Names Semgrep Supply Chain for open source dependency scanning and malware blocking, and Semgrep Secrets for hardcoded credentials found via semantic analysis. Keeps the two products distinct and does not attribute either capability to Semgrep Code.

Semgrep Secrets

Detection and remediation of hardcoded credentials using semantic analysis, positioned to prevent credential exposure and leaks.

“Find and fix hardcoded secrets with semantic analysis” semgrep.dev

Mapped capabilities

3 capabilities

Hardcoded secret detection
Recognizing Semgrep Secrets as the surface for credentials committed into code.
Semantic analysis differentiation
Explaining the semantic-analysis framing versus generic pattern matching, without inventing detection mechanics.
Credential exposure remediation
Describing the find-and-fix framing for leaked secrets.

AI-Assisted Detection, Triage & Remediation

The Multimodal layer combining AI reasoning with rule-based analysis, and Semgrep Guardian for scanning and fixing AI-generated code at the moment it is written.

“Scan and fix AI-generated code the moment it's written” semgrep.dev

Mapped capabilities

4 capabilities

Guardian for AI-generated code
Routing questions about AI coding assistants and AI-written code to Guardian.
Multimodal AI + rules
Explaining that AI reasoning is combined with rule-based analysis rather than replacing it.
AI-assisted triage
Describing where AI assists triage of findings across the platform.
AI-assisted remediation
Describing fix assistance without promising unstated autofix guarantees.

Illustrative example

Input: Our team is rolling out an AI coding assistant. Does Semgrep have anything that checks AI-written code before it lands?
Expected behavior: Identifies Semgrep Guardian as the product that scans and fixes AI-generated code the moment it is written, and may note that Multimodal combines AI reasoning with rule-based analysis. Does not claim named integrations with specific coding assistants.

AppSec Platform, Workflows & Integrations

The unifying platform layer: org-wide security management and enforcement, Semgrep Workflows for building security pipelines that combine static analysis with AI at scale, and the CI/developer-tool integration surface.

“Combine AI reasoning with rule-based analysis for detection, triage, and remediation” semgrep.dev

Mapped capabilities

4 capabilities

Org-wide policy enforcement
Explaining how the AppSec Platform automates, manages, and enforces security across an organization.
Semgrep Workflows
Describing building and deploying security pipelines that combine static analysis with AI at scale.
CI and developer-tool integrations
Answering which integrations exist, grounded in the integrations page rather than assumed vendors.
Secure guardrails in the dev flow
Mapping the secure guardrails solution framing onto developer workflow surfaces.

Plans, Packaging & Migration

How the product line is packaged and adopted: pricing and plans, the free/Community Edition entry points, the ROI calculator, and the New Shift Program for teams moving off Checkmarx or Snyk.

“The New Shift Program helps you switch from Checkmarx or Snyk” semgrep.dev

Mapped capabilities

4 capabilities

Plan and product packaging
Explaining which products are separately named and where pricing information lives.
Free and Community Edition entry points
Distinguishing the try-for-free and Community Edition paths from paid platform tiers.
New Shift migration program
Identifying the program for switching from Checkmarx or Snyk.
ROI calculator and evaluation support
Pointing buyers to the ROI calculator, docs, case studies, and demo booking.

Coverage is mapped from Semgrep's public pages (8 crawled). Examples are illustrative, not real test cases. The runnable eval library — graded inputs, expected behavior, and pass/fail checks — is built when you request it above.

Frequently asked questions

What do the Corsac evals for Semgrep test?+

The coverage map is generated from Semgrep's own public product surface (application security (AppSec) platform — SAST, SCA, and secrets detection): 6 scoring areas — Semgrep Code (SAST), Supply Chain & Malware Protection, and Semgrep Secrets, and more — spanning 22 mapped capabilities, each graded on adversarial robustness, workflow quality, safety gates, and operator quality once the library is built.

How are the Semgrep evals scored?+

Every case generated for Semgrep — across Semgrep Code (SAST) and Supply Chain & Malware Protection and the other mapped areas — is graded with pass/fail checks plus an LLM judge scoring 1–5 against its expected behavior, with critical-severity flags and negative controls. Only judge-passed evals are published.

How many test cases does the Semgrep library include?+

The full Semgrep library is built on request. The coverage map spans 6 areas and 22 capabilities (for example, Source-code vulnerability detection and Pro Engine language coverage under Semgrep Code (SAST)); each becomes graded test cases — inputs, expected behavior, pass/fail checks — in your Corsac workspace.

How do I run these evals against Semgrep or my own agent?+

Request the library with your work email above. We'll build out all 6 mapped Semgrep areas and set them up in a Corsac workspace, where you can run every test case against Semgrep or your own agent with your own data.