Eval directory · Security Operations

Evals for Socket

Eval coverage for Socket, mapped from its public product surface.

About Socket

Socket is a developer security platform that scans open source dependencies across major package registries for malicious behavior and blocks them before they reach production code. It ships as a CLI firewall at install time (Socket Firewall) and a GitHub App that flags risky dependency additions and malicious updates in pull requests. Paid tiers add precomputed reachability analysis to reduce CVE false positives, priority scoring, SBOM import/export, SSO/SAML, and compliance integrations.

Industry

open source software supply chain security

Website

socket.dev

Use the eval library for Socket

We'll build out the full library — runnable test cases with inputs, expected behavior, and pass/fail checks — in your Corsac workspace.

Related in Security Operations

All evals →

More Security Operations eval libraries

Coverage map

What would you measure for Socket?

6 scoring areas · 24 capabilities mapped · grounded in 8 cited pages

Every eval set is graded on

Adversarial robustness
Workflow quality
Safety gates
Operator quality

Pass/Fail + LLM judge 1–5 · critical severity flags · negative controls

Install-time blocking (Socket Firewall)

The CLI firewall that wraps package installs (e.g. `sfw npm install`) and blocks malicious packages before they land in the local dependency tree, reporting what was installed versus blocked.

“Socket blocks malicious packages before they reach your code.” socket.dev

Mapped capabilities

4 capabilities

Block malicious package at install
Detecting a package flagged as malicious during an install run and preventing it from being installed while allowing clean packages through.
Install summary and exit reporting
Reporting per-package outcomes and an accurate installed/blocked tally at the end of a wrapped install.
Multi-registry install coverage
Applying install-time scanning across the registries named in context: npm, PyPI, RubyGems, and Go.
Transitive dependency interception
Catching malicious code introduced through a transitive dependency rather than a directly requested package.

Illustrative example

Input: Run `sfw npm install` on a project depending on lodash@4.17.21, express@4.18.2, colors@1.4.1, and react@18.2.0, where colors@1.4.1 contains known malicious code.
Expected behavior: The three clean packages install and colors@1.4.1 is blocked with a stated reason of detected malicious code. The run ends with a summary reporting three packages installed and one threat blocked.

Pull request review (Socket for GitHub)

The GitHub App that inspects dependency changes in pull requests and surfaces risky additions and malicious updates before merge, with per-alert Block or Warn actions.

“Socket scans every package and update for malicious behavior across all major registries.” socket.dev

Mapped capabilities

4 capabilities

Flag risky dependency additions
Detecting newly added dependencies in a PR that carry known malware, typosquat, or install-script risk.
Detect malicious version updates
Identifying a version bump of an existing dependency that introduces malicious behavior.
Block versus Warn action assignment
Assigning the correct enforcement action and severity to each alert as presented in the PR alert table.
PR alert presentation
Rendering alerts with the package ecosystem, package name, alert type, and action in a form a reviewer can act on before merge.

Illustrative example

Input: A pull request adds npm package bowserify and pnpm package legacy-build-tool, one a suspected typosquat of a popular package and one declaring an HTTP dependency.
Expected behavior: The PR review surfaces two separate alerts before merge: bowserify as a potential typosquat with action Block, and legacy-build-tool as an HTTP dependency with action Warn, each labeled with its ecosystem and alert type.

Threat and risk detection coverage

The underlying detection catalog spanning 70+ risk types, including malware, typosquats, install scripts, supply chain exfiltration behavior, vulnerabilities, and license risk, plus AI analysis of hidden dependency behavior.

“Detect 70+ risk types (malware, vulnerabilities, license, etc.)” socket.dev

Mapped capabilities

4 capabilities

Malware and supply chain attack detection
Identifying packages that execute malicious payloads or exfiltrate data such as environment variables.
Typosquat identification
Recognizing packages whose names impersonate popular packages, as in the loadash and bowserify cases.
Install script and HTTP dependency risk
Flagging install-time scripts and dependencies fetched over insecure transports as distinct risk types.
AI analysis of hidden behavior
Surfacing dependency behavior that static risk lists alone would not flag, as described in the Free tier feature set.

Reachability and alert prioritization

Precomputed reachability analysis and priority scoring that reduce CVE false positives and direct attention to alerts that matter, with function-level reachability positioned at the Enterprise tier.

“precomputed reachability analysis cuts 60% of CVE false positives automatically, no extra setup needed” socket.dev

Mapped capabilities

4 capabilities

Reachable versus unreachable classification
Labeling a finding by whether the vulnerable code is reachable, potentially reachable, or unreachable in the consuming application.
CVE false positive reduction
Suppressing or de-emphasizing CVE alerts that reachability analysis rules out, consistent with the advertised 60% reduction.
Priority scoring
Ordering alerts by criticality and exposure so the highest-risk findings surface first.
Direct versus transitive and production context
Attributing each alert to direct or transitive dependency status and to production usage.

Plans, entitlements, and quotas

The Free, Team, Business, and Enterprise tiers with their stated seat minimums, scan and API quotas, repository label limits, and feature gates.

Mapped capabilities

4 capabilities

Scan and API quota enforcement
Applying the per-plan monthly scan limits and hourly API quotas stated in pricing.
Seat minimums and member limits
Enforcing the 5-developer Team and 20-developer Business minimums and the member limits per tier.
Feature gating by tier
Restricting reachability, priority scoring, SBOM, SSO/SAML, and compliance integrations to the tiers that include them.
Repository label limits
Applying the 1, 3, and unlimited repository label allowances across Free, Team, and Business.

Org administration, integrations, and compliance

Business and Enterprise administrative surfaces: SSO/SAML, webhook automation, Slack alerting, SBOM import/export, compliance integrations such as Vanta, and scanning of GitHub Actions and AI models.

“Compliance integrations (e.g. Vanta)” socket.dev

Mapped capabilities

4 capabilities

SBOM import and export
Producing and ingesting SBOMs for full dependency visibility as a Business-tier capability.
SSO/SAML and access administration
Configuring single sign-on and managing organization membership at the Business tier.
Alerting and webhook automation
Delivering Slack alerts for new malware or vulnerabilities and driving webhook-based automation.
GitHub Actions and AI model scanning
Extending scanning beyond package dependencies to GitHub Actions workflows and AI models.

Coverage is mapped from Socket's public pages (8 crawled). Examples are illustrative, not real test cases. The runnable eval library — graded inputs, expected behavior, and pass/fail checks — is built when you request it above.

Frequently asked questions

What do the Corsac evals for Socket test?+

The coverage map is generated from Socket's own public product surface (open source software supply chain security): 6 scoring areas — Install-time blocking (Socket Firewall), Pull request review (Socket for GitHub), and Threat and risk detection coverage, and more — spanning 24 mapped capabilities, each graded on adversarial robustness, workflow quality, safety gates, and operator quality once the library is built.

How are the Socket evals scored?+

Every case generated for Socket — across Install-time blocking (Socket Firewall) and Pull request review (Socket for GitHub) and the other mapped areas — is graded with pass/fail checks plus an LLM judge scoring 1–5 against its expected behavior, with critical-severity flags and negative controls. Only judge-passed evals are published.

How many test cases does the Socket library include?+

The full Socket library is built on request. The coverage map spans 6 areas and 24 capabilities (for example, Block malicious package at install and Install summary and exit reporting under Install-time blocking (Socket Firewall)); each becomes graded test cases — inputs, expected behavior, pass/fail checks — in your Corsac workspace.

How do I run these evals against Socket or my own agent?+

Request the library with your work email above. We'll build out all 6 mapped Socket areas and set them up in a Corsac workspace, where you can run every test case against Socket or your own agent with your own data.