Eval directory

Evals for Pixee

Eval coverage for Pixee, mapped from its public product surface.

About Pixee

Pixee is an AI-native "resolution" platform that ingests findings from existing security scanners, triages them to isolate genuinely exploitable risk, and generates fixes developers can merge. It pairs reactive remediation (Pixee VulnOps) with proactive design-time review (Foresight) to catch risk before vulnerable code is generated. It is sold on outcome-based pricing tied to vulnerabilities resolved rather than developer seats, with self-hosted, air-gapped, and cloud deployment options.

Industry

agentic application security (AppSec) triage and remediation platform

Website

www.pixee.ai

Use the eval library for Pixee

We'll build out the full library — runnable test cases with inputs, expected behavior, and pass/fail checks — in your Corsac workspace.

All evals →

More AI eval libraries

Coverage map

What would you measure for Pixee?

6 scoring areas · 23 capabilities mapped · grounded in 8 cited pages

Every eval set is graded on

Adversarial robustness
Workflow quality
Safety gates
Operator quality

Pass/Fail + LLM judge 1–5 · critical severity flags · negative controls

Scanner Ingestion & Attack Surface Understanding

Consuming findings from the existing security stack and building an accurate picture of the real attack surface before any triage or fix work begins.

“We ingest findings from any scanner, triage, prioritize and fix what is truly exploitable.” www.pixee.ai

Mapped capabilities

4 capabilities

Multi-scanner finding ingestion
Accepts findings across SAST and SCA sources; behavior when scanner output is malformed, partial, or from an unsupported format.
Attack surface modeling
The 'Understand Your Real Attack Surface' step — assembling repository and dependency context that later stages depend on.
Source control integration
GitHub and GitLab connectivity, including repository access scope and behavior when permissions are insufficient.
Finding normalization and deduplication
Reconciling the same underlying issue reported by more than one scanner into a single unit of work.

Intelligent Triage & Reachability

Separating exploitable vulnerabilities from theoretical ones — the claimed 98%+ false-positive reduction — and justifying each verdict.

“Intelligent triage eliminates 98%+ of false positives” www.pixee.ai

Mapped capabilities

4 capabilities

Reachability analysis
Tracing code execution paths to establish whether a flagged sink is actually reachable from untrusted input.
False-positive suppression
Marking findings as not-exploitable without silently dropping genuinely risky ones; conservative behavior under ambiguity.
Exploitability prioritization
Ordering the surviving findings so the highest real risk surfaces first rather than by raw scanner severity.
Triage rationale
Producing an inspectable explanation a human reviewer can audit for each dismissal or escalation.

Illustrative example

Input: A SAST finding on a vulnerable sink in a module with no execution path from any untrusted input, submitted for triage.
Expected behavior: The finding is classified as not exploitable and the output includes a reachability rationale naming the entry points examined and why no path reaches the sink, rather than an unexplained dismissal.

Automated Fix Generation & Validation

Producing production-ready remediation code that developers merge, gated by the stated three validation layers.

“Automated remediation ships production ready code” www.pixee.ai

Mapped capabilities

4 capabilities

Fix synthesis for SAST and SCA findings
Generating a concrete code change that removes the vulnerability rather than describing one.
Three-layer validation
Syntax, test-suite, and policy-compliance checks before a fix is offered; behavior when a layer fails.
Merge-ready change packaging
Delivering the fix into code review in a form that turns fix authors into fix reviewers.
Non-regression and scope discipline
Keeping the change limited to the vulnerability at hand without unrelated refactors or behavior changes.

Pixee Context System (Codebase Fidelity)

Making fixes reflect how the team actually codes — using existing internal abstractions, patterns, and architectural constraints instead of generic advice.

Mapped capabilities

4 capabilities

Existing-abstraction reuse
Preferring the codebase's own safe helper (e.g. an in-repo SafeQueryBuilder) over a generic parameterized-query snippet.
Architectural constraint respect
Honoring structural constraints of the codebase when choosing a remediation strategy.
Team pattern and preference learning
Conforming to established conventions, policies, and preferences observed in the repository.
Idiomatic consistency across languages and frameworks
Fix style that matches the surrounding code rather than importing a foreign idiom.

Illustrative example

Input: SQL injection finding on a raw string-concatenated query in a repo that already defines and widely uses an internal SafeQueryBuilder class for parameterized access.
Expected behavior: The generated fix routes the query through the existing SafeQueryBuilder abstraction rather than introducing a standalone parameterized-query snippet, and the change compiles against the class's actual signature.

Foresight: Design-Time Prevention

Proactive review that catches risk in designs before vulnerable code is generated, forming the second half of the stated compounding loop.

Mapped capabilities

3 capabilities

Design review before code generation
Identifying risk in a proposed design at the pre-implementation stage.
Preventive guidance quality
Recommendations specific enough to change the resulting implementation rather than restating generic secure-design principles.
Loop with reactive remediation
Feeding what VulnOps resolves back into what Foresight prevents, and vice versa.

Enterprise Deployment, Policy & Outcomes

The governance envelope enterprises buy against: deployment model, custom policy, audit evidence, and outcome-based pricing.

“Production ready code that passes your tests. Runs three validation layers: syntax, tests, and policy compliance.” www.pixee.ai

Mapped capabilities

4 capabilities

Deployment model handling
Self-hosted, air-gapped, and cloud options, including behavior under air-gapped constraints.
Custom security policies
Applying customer-defined policy as a gate on which fixes are generated and shipped.
Compliance audit trails
Producing the trail of triage and remediation decisions referenced for SOC2 and ISO27001 posture.
Outcome-based pricing and ROI representation
Accurately reflecting the pay-per-vulnerability-resolved model and the ROI calculator's assumption inputs.

Coverage is mapped from Pixee's public pages (8 crawled). Examples are illustrative, not real test cases. The runnable eval library — graded inputs, expected behavior, and pass/fail checks — is built when you request it above.

Frequently asked questions

What do the Corsac evals for Pixee test?+

The coverage map is generated from Pixee's own public product surface (agentic application security (AppSec) triage and remediation platform): 6 scoring areas — Scanner Ingestion & Attack Surface Understanding, Intelligent Triage & Reachability, and Automated Fix Generation & Validation, and more — spanning 23 mapped capabilities, each graded on adversarial robustness, workflow quality, safety gates, and operator quality once the library is built.

How are the Pixee evals scored?+

Every case generated for Pixee — across Scanner Ingestion & Attack Surface Understanding and Intelligent Triage & Reachability and the other mapped areas — is graded with pass/fail checks plus an LLM judge scoring 1–5 against its expected behavior, with critical-severity flags and negative controls. Only judge-passed evals are published.

How many test cases does the Pixee library include?+

The full Pixee library is built on request. The coverage map spans 6 areas and 23 capabilities (for example, Multi-scanner finding ingestion and Attack surface modeling under Scanner Ingestion & Attack Surface Understanding); each becomes graded test cases — inputs, expected behavior, pass/fail checks — in your Corsac workspace.

How do I run these evals against Pixee or my own agent?+

Request the library with your work email above. We'll build out all 6 mapped Pixee areas and set them up in a Corsac workspace, where you can run every test case against Pixee or your own agent with your own data.