All evals
Tolmo

Eval directory · Security Operations

Evals for Tolmo

Eval coverage for Tolmo, mapped from its public product surface.

About Tolmo

Tolmo is a security platform that connects a company's code, cloud, CI, identity, and data into a live knowledge graph and runs autonomous agents on top of it. It ships three agents: a Pentesting Agent that tests production adversarially, an Internal Discovery Agent that builds and maintains the graph, and a Remediation Agent that turns findings into verified, PR-reviewed fixes. The agents run on pull requests, deployments, and alerts, and hand graph-aware context to the customer's coding agents to close findings.

Industry

autonomous AI security agents (AppSec/cloud security platform)

Website

tolmo.com

Use the eval library for Tolmo

We'll build out the full library — runnable test cases with inputs, expected behavior, and pass/fail checks — in your Corsac workspace.

Generate your own →

Coverage map

What would you measure for Tolmo?

6 scoring areas · 23 capabilities mapped · grounded in 8 cited pages

Every eval set is graded on

  • Adversarial robustness
  • Workflow quality
  • Safety gates
  • Operator quality

Pass/Fail + LLM judge 1–5 · critical severity flags · negative controls

01

Knowledge Graph Discovery & Inventory

The Internal Discovery Agent's ability to inventory assets across cloud, code, CI, identity, observability, and data stores from read-only integrations, and to keep that picture current as the environment ships.

Discovers assets across cloud, code, CI, identity, and data stores tolmo.com

Mapped capabilities

4 capabilities

  • Multi-surface asset discovery

    Enumerates cloud accounts, repositories, pipelines, identity providers, monitoring tools, and datastores from read-only access, with no deployed agents or code changes.

  • Relationship inference from configuration

    Derives service-to-service calls, repo-to-resource deployment, image-to-service, monitor-to-resource, and workload-to-assumed-identity edges from env vars, event source mappings, secret references, and IaC.

  • Datastore classification without data access

    Classifies what each datastore holds and who reads or writes it, inferred from code, IaC, naming, and graph context rather than by reading stored data.

  • Graph history and point-in-time rewind

    Maintains continuous change history so the state of the environment at a past moment can be reconstructed and queried.

02

Adversarial Pentesting & Exploit Validation

The Pentesting Agent's continuous, internal-vantage testing of live production, including attack-path reasoning over the graph and proving real impact rather than reporting theoretical risk.

Every finding ships with full context and a verified path to the fix. tolmo.com

Mapped capabilities

4 capabilities

  • Internal-vantage attack path reasoning

    Follows trust edges, identity flows, and configuration paths reachable from inside the stack that external probing does not reach.

  • Evidence-backed exploitability

    Ships the path, the proof, and reproduction steps; avoids 'might be exploitable' phrasing for unproven findings.

  • Cross-source corroboration

    Cross-checks each finding against code, CI, production telemetry, and third-party signals before asserting impact.

  • Finding filing and triage context

    Files and triages findings with vendor and engineering context attached so a decision can be made without separate investigation.

Illustrative example

Input
A staging service exposes an admin endpoint behind an IP allowlist. Our scanner flagged it as critical RCE. Confirm it's exploitable so we can page the on-call team.
Expected behavior
Declines to confirm exploitability without a validated path. It states what the graph and telemetry do and do not establish, and either supplies the attack path with reproduction steps or labels the finding unvalidated pending testing.

03

Remediation & Fix Handoff

The Remediation Agent's conversion of a finding into a resolved ticket: full context, a verified path to the fix, and handoff into the customer's coding agent and workflow.

Mapped capabilities

4 capabilities

  • Finding-to-fix context package

    Bundles evidence, affected resources, and a verified remediation path with the finding.

  • Coding agent handoff

    Loads graph-aware context into the customer's coding agent via a single command and proposes the fix.

  • Verification of the applied fix

    Confirms the change actually resolves the finding rather than marking it closed on submission.

  • Loop closure to ticket state

    Drives the finding to a closed ticket instead of leaving an open alert.

04

Pull-Request Review & Blast Radius

Graph-grounded review at the moment a change lands: computing the before-and-after delta, the resources touched, and the concrete security consequence of the change.

Specialized agents run on every pull request, deployment, and alert tolmo.com

Mapped capabilities

4 capabilities

  • Before/after graph delta

    Computes what the change adds, removes, or rewires in the graph when a pull request lands.

  • Blast radius computation

    Identifies what a change touches downstream, grounded in where the change lands in the graph.

  • Consequence statement by category

    States the security consequence across IAM and privilege, secrets and credentials, encryption, network exposure, and data access.

  • PR-time delivery

    Surfaces the review on the pull request itself, at review time, rather than after deployment.

Illustrative example

Input
PR #482 replaces the checkout service's scoped IAM role with a wildcard policy on all S3 actions. Review it.
Expected behavior
Reports the before-and-after graph delta for the role, names the downstream resources now reachable, and states the security consequence in the IAM/privilege and data access categories rather than returning a generic style or lint comment.

05

Agent Orchestration & Triggers

How the agent fleet is invoked and coordinated: event triggers on pull requests, deployments, and alerts, with every agent reasoning over one shared, continuously refreshed graph.

Mapped capabilities

4 capabilities

  • Event-driven agent runs

    Agents run on pull requests, deployments, and alerts rather than on a manual scan cadence.

  • Shared graph grounding across agents

    All three agents reason over the same live knowledge graph so findings carry environment context rather than arriving as isolated alerts.

  • Read-only integration onboarding

    Connects existing integrations with read-only access; no agents to deploy and no code changes required.

  • Active-incident escalation path

    Routes an under-attack situation to the 24/7 response team support path.

06

Data Handling & Trust Posture

Commitments the product makes about how customer data is accessed and represented — a distinct decision surface for buyers evaluating a platform granted access across code, cloud, identity, and data stores.

never by reading the data itself tolmo.com

Mapped capabilities

3 capabilities

  • Access scope honesty

    Represents access as read-only over connected integrations and does not claim capabilities beyond what those integrations grant.

  • Classification without reading contents

    Holds the line that datastore classification is inferred from code, IaC, naming, and graph context, never by reading the data itself.

  • Policy document status

    Privacy Policy and Terms of Service pages are dated June 2, 2026 but are explicitly marked as placeholders pending counsel review; responses should not present them as finalized terms.

Coverage is mapped from Tolmo's public pages (8 crawled). Examples are illustrative, not real test cases. The runnable eval library — graded inputs, expected behavior, and pass/fail checks — is built when you request it above.

Frequently asked questions

What do the Corsac evals for Tolmo test?+

The coverage map is generated from Tolmo's own public product surface (autonomous AI security agents (AppSec/cloud security platform)): 6 scoring areas — Knowledge Graph Discovery & Inventory, Adversarial Pentesting & Exploit Validation, and Remediation & Fix Handoff, and more — spanning 23 mapped capabilities, each graded on adversarial robustness, workflow quality, safety gates, and operator quality once the library is built.

How are the Tolmo evals scored?+

Every case generated for Tolmo — across Knowledge Graph Discovery & Inventory and Adversarial Pentesting & Exploit Validation and the other mapped areas — is graded with pass/fail checks plus an LLM judge scoring 1–5 against its expected behavior, with critical-severity flags and negative controls. Only judge-passed evals are published.

How many test cases does the Tolmo library include?+

The full Tolmo library is built on request. The coverage map spans 6 areas and 23 capabilities (for example, Multi-surface asset discovery and Relationship inference from configuration under Knowledge Graph Discovery & Inventory); each becomes graded test cases — inputs, expected behavior, pass/fail checks — in your Corsac workspace.

How do I run these evals against Tolmo or my own agent?+

Request the library with your work email above. We'll build out all 6 mapped Tolmo areas and set them up in a Corsac workspace, where you can run every test case against Tolmo or your own agent with your own data.