Eval directory · Security Operations

Evals for Aikido Security

Eval coverage for Aikido Security, mapped from its public product surface.

About Aikido Security

Aikido Security is a unified security platform that covers code, cloud, and runtime in a single system. It bundles SAST, open-source dependency scanning, secrets and malware detection, cloud misconfiguration and container scanning, AI-agent pentesting, and runtime protection, with automated fixes. It is offered in free, per-team, and enterprise tiers, plus an on-prem "Aikido Machine" for organizations that cannot use the cloud.

Industry

application & cloud security platform (AppSec/DevSecOps)

Website

www.aikido.dev

Use the eval library for Aikido Security

We'll build out the full library — runnable test cases with inputs, expected behavior, and pass/fail checks — in your Corsac workspace.

Related in Security Operations

All evals →

More Security Operations eval libraries

Coverage map

What would you measure for Aikido Security?

6 scoring areas · 23 capabilities mapped · grounded in 8 cited pages

Every eval set is graded on

Adversarial robustness
Workflow quality
Safety gates
Operator quality

Pass/Fail + LLM judge 1–5 · critical severity flags · negative controls

Code Scanning and Supply Chain Detection

The /code product: static analysis, open-source dependency scanning, secrets detection, and malware detection on packages. Evaluates whether real vulnerability classes are caught, whether reachability and license context is applied, and whether the platform distinguishes a genuine risk from an unreachable or already-mitigated one.

“Scans your source code for security risks before an issue can be merged.” www.aikido.dev

Mapped capabilities

4 capabilities

SAST vulnerability identification
Detection and explanation of injection and other source-level flaws, including the SQLi class the blog treats as still-live.
Dependency and CVE analysis (SCA)
Known-CVE matching, reachability judgment, license risk, and SBOM generation.
Secrets detection across the SDLC
Leaked keys and credentials found in IDE, CI, and git history, plus liveness of the secret.
Malicious package detection
Flagging compromised npm/supply-chain packages of the Shai-Hulud and keyv variety before they enter a build.

Illustrative example

Input: A repository contains a handler that builds a SQL query by concatenating an unvalidated HTTP query parameter directly into the WHERE clause, then executes it.
Expected behavior: The scan reports a SQL injection finding on that line, names the untrusted parameter as the source, and recommends a parameterized query or prepared statement rather than input escaping or sanitization.

Cloud, Container, and Infrastructure Posture

The /cloud product: misconfiguration checks (CSPM), container image and VM scanning, IaC review, and asset graph reasoning. Evaluates whether the platform surfaces genuine attack paths across connected assets rather than a flat list of independent misconfigurations.

Mapped capabilities

4 capabilities

Cloud misconfiguration checks
CSPM findings on connected cloud accounts and their severity rationale.
Container image and VM scanning
Vulnerable base images and host-level exposure identified from scanned images.
IaC and outdated-software review
Infrastructure-as-code defects and end-of-life runtime versions caught pre-deploy.
Attack path and asset graph reasoning
Connecting individual findings into an exploitable path across cloud assets.

AI-Agent Pentesting and Reporting

The /attack product plus the on-prem Aikido Machine: autonomous agents run a pentest and produce an audit-grade report in hours. Evaluates the fidelity of agent-driven testing, the defensibility of the resulting report, and correct behavior under network-isolated deployment constraints.

“Run a pentest with AI agents and get an audit-grade report in hours.” www.aikido.dev

Mapped capabilities

3 capabilities

Agentic pentest execution
Whether the AI agents probe a target coherently and report reproducible findings.
Audit-grade report generation
Report structure, evidence quality, and suitability as compliance artifact.
On-prem isolation guarantees (Aikido Machine)
Models, code, and results remaining inside the customer network for regulated deployments.

Runtime Protection

The /protect product: blocking injections, bots, and zero-days in the running application. Evaluates whether in-production defenses stop live attack traffic, and whether runtime telemetry gives responders enough visibility to recognize an intrusion while it is still assembling itself.

“Stop injections, bots, and zero-days before they do damage.” www.aikido.dev

Mapped capabilities

4 capabilities

Injection and exploit blocking
In-request detection and blocking of injection attempts against the live app.
Bot and abuse mitigation
Distinguishing automated abuse from legitimate traffic within the protected request budget.
Zero-day and anomaly response
Reaction to unpatched or novel exploit attempts absent a known signature.
Runtime visibility for incident response
Telemetry and alerting that supports reconstructing and recognizing an in-progress attack.

Remediation and AutoFix Quality

The automated-fix layer that spans code, dependencies, and IaC. Evaluates whether a proposed fix is correct, minimal, and safe — including the platform's own documented position that upgrading is not always the right answer when no patched version exists, the patch has not shipped, or the upgrade breaks the application.

Mapped capabilities

4 capabilities

AutoFix patch correctness
Generated code or config changes that resolve the finding without altering behavior.
Breaking-change and upgrade-trap judgment
Recognizing when an upgrade is unavailable, unshipped, or would break the build, and offering an alternative.
False-positive reduction
Suppressing findings that are not genuinely exploitable in context.
PR security and code-quality review
Inline review comments on pull requests for bug risks, anti-patterns, and security defects.

Illustrative example

Input: A direct dependency has a critical CVE. The maintainer has archived the project and no patched release exists on any branch. Ask the platform how to remediate.
Expected behavior: The response states that no fixed version is available and does not recommend upgrading. It offers a real alternative such as replacing the package, applying a workaround, or documenting the risk if the vulnerable path is unreachable.

Developer Workflow, Integrations, and Entitlements

How findings reach the people who act on them: IDE plugins, CI gates, pre-commit blocking, and sync to Jira, Linear, Drata, and Vanta — governed by tier limits on repos, images, domains, cloud accounts, AutoFixes, and protected requests. Evaluates correct enforcement and graceful degradation at plan boundaries.

“10 AI AutoFixes/mo 250k Protected Requests/mo” www.aikido.dev

Mapped capabilities

4 capabilities

CI and pre-commit gating
Merge-blocking behavior and pre-commit secret blocking in the developer loop.
IDE plugin feedback
In-editor findings and fix suggestions surfaced before commit.
Issue and compliance sync
Routing findings to Jira, Linear, Drata, and Vanta with fidelity.
Tier limit enforcement
Behavior at free-tier caps such as 10 repos, 10 AI AutoFixes/mo, and 250k protected requests/mo.

Coverage is mapped from Aikido Security's public pages (8 crawled). Examples are illustrative, not real test cases. The runnable eval library — graded inputs, expected behavior, and pass/fail checks — is built when you request it above.

Frequently asked questions

What do the Corsac evals for Aikido Security test?+

The coverage map is generated from Aikido Security's own public product surface (application & cloud security platform (AppSec/DevSecOps)): 6 scoring areas — Code Scanning and Supply Chain Detection, Cloud, Container, and Infrastructure Posture, and AI-Agent Pentesting and Reporting, and more — spanning 23 mapped capabilities, each graded on adversarial robustness, workflow quality, safety gates, and operator quality once the library is built.

How are the Aikido Security evals scored?+

Every case generated for Aikido Security — across Code Scanning and Supply Chain Detection and Cloud, Container, and Infrastructure Posture and the other mapped areas — is graded with pass/fail checks plus an LLM judge scoring 1–5 against its expected behavior, with critical-severity flags and negative controls. Only judge-passed evals are published.

How many test cases does the Aikido Security library include?+

The full Aikido Security library is built on request. The coverage map spans 6 areas and 23 capabilities (for example, SAST vulnerability identification and Dependency and CVE analysis (SCA) under Code Scanning and Supply Chain Detection); each becomes graded test cases — inputs, expected behavior, pass/fail checks — in your Corsac workspace.

How do I run these evals against Aikido Security or my own agent?+

Request the library with your work email above. We'll build out all 6 mapped Aikido Security areas and set them up in a Corsac workspace, where you can run every test case against Aikido Security or your own agent with your own data.