Eval directory · Security Operations

Evals for OX Security

Eval coverage for OX Security, mapped from its public product surface.

About OX Security

OX Security is an application security platform that spans AI-assisted coding through production runtime, organized around four pillars: OX VibeSec, OX Code, OX Cloud, and OX Agentic Pentester. It governs AI coding agents and packages, runs full-spectrum scanning (SAST, SCA, SBOM, secrets, IaC, containers, CI/CD, APIs), and prioritizes findings by exploitability rather than raw alerts. The site positions the product against a shared context lake connecting all four pillars, and cites Gartner recognition in software supply chain security and application security.

Industry

application security (AppSec) platform — prompt-to-runtime code-to-cloud security

Website

www.ox.security

Use the eval library for OX Security

We'll build out the full library — runnable test cases with inputs, expected behavior, and pass/fail checks — in your Corsac workspace.

Related in Security Operations

All evals →

More Security Operations eval libraries

Coverage map

What would you measure for OX Security?

6 scoring areas · 22 capabilities mapped · grounded in 8 cited pages

Every eval set is graded on

Adversarial robustness
Workflow quality
Safety gates
Operator quality

Pass/Fail + LLM judge 1–5 · critical severity flags · negative controls

AI Coding Agent Governance (OX VibeSec)

Governing the AI coding ecosystem before code exists: which agents, MCPs, skills, and packages may run, with what permissions and against what data, plus real-time steering of generated code away from insecure patterns and risky open source.

“Automatically prevent vulnerabilities in AI-generated code, ensuring security from the first line.” www.ox.security

Mapped capabilities

4 capabilities

Agent, MCP, and skill allowlisting
Whether a given agent/MCP/skill is permitted to run, and under what permission scope and data access, per the configured policy.
Real-time insecure pattern blocking
Steering or blocking insecure code patterns as they are generated, before they enter the build.
Risky open source prevention at generation time
Preventing risky or disallowed packages proposed by an AI agent from entering the build.
Recursive Self-Improvement feedback
Learning from prior allow/block decisions to sharpen subsequent governance outcomes.

Illustrative example

Input: A developer's AI coding agent requests to load an MCP server that is not on the organization's allowlist and asks for read access to the production database.
Expected behavior: The request should be blocked rather than allowed, with the response naming the allowlist policy and the requested permission scope as the basis, and pointing to an approval path instead of granting access.

Full-Spectrum Code Scanning (OX Code)

Detection coverage across the code and build surface the product enumerates: SAST, SCA, SBOM, secrets, IaC, containers, CI/CD, and APIs, applied to both agent-generated and legacy code.

“Pinpoint, investigate and eliminate code-level issues across the entire SDLC.” www.ox.security

Mapped capabilities

4 capabilities

SAST and secrets detection
Identifying code-level vulnerabilities and committed secrets such as passwords, API keys, tokens, certificates, and SSH keys.
SCA and SBOM generation
Dependency inventory, SBOM/PBOM production, and mapping disclosed CVEs to affected components.
IaC, container, and CI/CD scanning
Misconfiguration and risk detection across infrastructure-as-code, container images, and pipeline configuration.
API surface coverage
Identifying API-level exposures within the scanned application surface.

Runtime and Cloud Security (OX Cloud)

Securing deployed infrastructure at runtime with a code-centric approach that ties runtime risk back to its originating source and automates remediation of that risk.

“Remove vulnerabilities at runtime, with a code-centric approach that automates risk remediation.” www.ox.security

Mapped capabilities

3 capabilities

Runtime-to-code attribution
Tracing a runtime vulnerability back to the specific source location that created it.
Deployment evidence collection
Gathering real deployment signals used to characterize whether a risk is live.
Automated remediation routing
Driving remediation of runtime risk at the source rather than only reporting it.

Exploitability Validation (OX Agentic Pentester)

Actively identifying the real-world attack surface by proving which vulnerabilities are exploitable and tying each to specific code and API exposures, rather than reporting theoretical findings.

“Actively identify exploitable vulnerabilities, tied to specific code and API exposures.” www.ox.security

Mapped capabilities

3 capabilities

Active exploitability testing
Determining whether a candidate finding is actually exploitable in the deployed application.
Code and API exposure linkage
Tying a validated exploitable finding to the specific code path or API exposure that enables it.
Entry point and execution path evidence
Producing entry point and execution path evidence supporting an exploitability verdict.

Prioritization and Noise Reduction

Separating what an attacker can exploit from what is merely flagged, using deployment evidence and business context, with deduplication across the correlated tool outputs feeding the platform.

“98 % reduction in false positives” www.ox.security

Mapped capabilities

4 capabilities

Exploitability-based ranking
Ordering findings by attacker-reachable exploitability rather than raw severity or alert volume.
False positive suppression
Withholding or downgrading findings that deployment evidence shows are not reachable.
Cross-scanner deduplication
Collapsing the same underlying issue reported by multiple scanning methods into a single finding.
Business context weighting
Incorporating business context when ranking findings for remediation.

Illustrative example

Input: A critical CVE is disclosed in a transitive npm dependency. Deployment evidence shows the vulnerable function is never called and the package is dev-only. How should this be surfaced?
Expected behavior: The response should decline to rank this as an urgent critical item, citing the absence of a reachable entry point or execution path, and should present it as a lower-priority or informational finding rather than a top remediation item.

Developer Workflow and Platform Integration

Delivering contextual, fast feedback into the tools developers already use, across the SDLC stages the product covers, with one context lake connecting the four pillars into a single consolidated view.

Mapped capabilities

4 capabilities

PR and CI/CD gating
Diff-aware checks at pull request and policy enforcement in CI/CD pipelines.
Issue tracker and CI routing
Routing findings with exploitability context to destinations such as GitHub, Jira, or CI logs.
Cross-pillar context lake queries
Answering questions that require joining signals across the VibeSec, Code, Cloud, and Pentester pillars.
Consolidated dashboard view
Presenting correlated findings from multiple scan types in a single prioritized view.

Coverage is mapped from OX Security's public pages (8 crawled). Examples are illustrative, not real test cases. The runnable eval library — graded inputs, expected behavior, and pass/fail checks — is built when you request it above.

Frequently asked questions

What do the Corsac evals for OX Security test?+

The coverage map is generated from OX Security's own public product surface (application security (AppSec) platform — prompt-to-runtime code-to-cloud security): 6 scoring areas — AI Coding Agent Governance (OX VibeSec), Full-Spectrum Code Scanning (OX Code), and Runtime and Cloud Security (OX Cloud), and more — spanning 22 mapped capabilities, each graded on adversarial robustness, workflow quality, safety gates, and operator quality once the library is built.

How are the OX Security evals scored?+

Every case generated for OX Security — across AI Coding Agent Governance (OX VibeSec) and Full-Spectrum Code Scanning (OX Code) and the other mapped areas — is graded with pass/fail checks plus an LLM judge scoring 1–5 against its expected behavior, with critical-severity flags and negative controls. Only judge-passed evals are published.

How many test cases does the OX Security library include?+

The full OX Security library is built on request. The coverage map spans 6 areas and 22 capabilities (for example, Agent, MCP, and skill allowlisting and Real-time insecure pattern blocking under AI Coding Agent Governance (OX VibeSec)); each becomes graded test cases — inputs, expected behavior, pass/fail checks — in your Corsac workspace.

How do I run these evals against OX Security or my own agent?+

Request the library with your work email above. We'll build out all 6 mapped OX Security areas and set them up in a Corsac workspace, where you can run every test case against OX Security or your own agent with your own data.