Eval directory

Evals for Ciroos

Eval coverage for Ciroos, mapped from its public product surface.

About Ciroos

Ciroos is an AI SRE "teammate" that integrates with enterprise observability, incident management, ticketing, and collaboration tools to investigate production disruptions. It correlates cross-domain telemetry and operational context — dependencies, changes, configurations, and historical behavior — to determine root cause rather than just surface alerts. The company positions it as a way to cut MTTR, reduce alert noise, and expand SRE capacity without adding headcount.

Industry

AI SRE agent for incident root cause analysis and observability

Website

ciroos.ai

Use the eval library for Ciroos

We'll build out the full library — runnable test cases with inputs, expected behavior, and pass/fail checks — in your Corsac workspace.

All evals →

More AI eval libraries

Coverage map

What would you measure for Ciroos?

6 scoring areas · 24 capabilities mapped · grounded in 8 cited pages

Every eval set is graded on

Adversarial robustness
Workflow quality
Safety gates
Operator quality

Pass/Fail + LLM judge 1–5 · critical severity flags · negative controls

Root Cause Investigation

Determining what actually caused a disruption using full operational context — dependencies, changes, configurations, and historical behavior — rather than restating which alerts fired.

“Ciroos investigates using full operational context — including dependencies, changes, configurations, and historical behavior” ciroos.ai

Mapped capabilities

4 capabilities

Cross-domain evidence correlation
Joining telemetry from cloud, infrastructure, application, and platform domains into one causal account.
Change and deployment attribution
Linking a disruption to the specific config change, deploy, or rollout that preceded it.
Dependency and blast-radius reasoning
Tracing upstream cause to downstream impact across service dependency paths.
Historical recurrence matching
Recognizing when an incident repeats a previously seen failure pattern.

Illustrative example

Input: Checkout latency alert at 14:02, ingress config change merged at 13:54, node pool autoscale at 13:58, and a downstream payments timeout alert at 14:03.
Expected behavior: Identifies the ingress config change as the probable root cause, traces the dependency path to the payments timeout as downstream impact, and marks the autoscale event as correlated but not causal.

Signal Intelligence and Noise Reduction

Analyzing raw cross-domain telemetry to cut alert volume beyond basic grouping, as claimed for Ciroos Signal Intelligence and the 70% noise reduction case study.

“Move beyond basic alert grouping.” ciroos.ai

Mapped capabilities

4 capabilities

Alert-to-incident consolidation
Collapsing many symptom alerts arising from one origin event into a single incident.
Separating co-occurring unrelated events
Avoiding over-merging of alerts that share timing but not cause.
Suppression without signal loss
Reducing noise while preserving alerts that represent genuine independent failures.
Severity and priority assessment
Ranking consolidated incidents by impact rather than by alert count.

Illustrative example

Input: Forty alerts in four minutes: thirty-six across six services tracing to one database failover, and four from an unrelated CDN cache-purge error in a different region.
Expected behavior: Groups the thirty-six alerts into a single incident attributed to the database failover, and keeps the four CDN alerts as a separate incident rather than merging or suppressing them as noise.

Tool Integration and Workflow Automation

Operating inside the observability, incident management, ticketing, and collaboration stack customers already use, including the automated ticketing described in the EV manufacturer case study.

“It brings deeper reasoning and context to the observability tools your SRE team already relies on.” ciroos.ai

Mapped capabilities

4 capabilities

Observability data retrieval
Pulling the right metrics, logs, and traces from connected sources during an investigation.
Incident management synchronization
Reflecting investigation state into the incident record without contradicting it.
Ticket creation and update automation
Filing and updating tickets with accurate scope, owner, and evidence.
Collaboration channel updates
Posting investigation status into team channels at the right granularity.

Investigation Communication and Handoff

Producing an account of the incident that a responding human can act on and that survives after the war room ends — the institutional-knowledge retention Ciroos positions as core.

Mapped capabilities

4 capabilities

Incident timeline narrative
Ordered account of what happened and why, distinguishing cause from symptom.
Evidence citation and confidence
Grounding each conclusion in named telemetry or change records with stated certainty.
Escalation recommendation
Identifying which team or expert is actually needed, avoiding misdirected escalation.
Post-incident knowledge capture
Retaining findings so a recurrence is resolved faster than the first occurrence.

Data Handling and Action Policy

Respecting the processor-scoped role and customer data boundaries stated in the Ciroos privacy policy, and constraining what the agent may write or execute in production systems.

“we generally do so as a service provider or processor on that Customer's behalf” ciroos.ai

Mapped capabilities

4 capabilities

Customer telemetry scope boundaries
Using operational data only within the customer context it was provided for.
Personal information in logs and tickets
Handling PII encountered in telemetry when summarizing or filing externally.
Least-privilege write actions
Limiting ticketing and remediation writes to authorized scope.
Auditability of agent actions
Making every automated action attributable and reviewable after the fact.

Uncertainty and Failure Recovery

Behavior when the operational picture is incomplete or contradictory — the condition under which a wrong confident answer causes the misdirected action and unnecessary escalation Ciroos claims to eliminate.

“Ciroos now identifies root causes in under 10 minutes and automates ticketing” ciroos.ai

Mapped capabilities

4 capabilities

Degraded or missing data sources
Continuing usefully when an observability or ticketing integration is unavailable.
Insufficient-evidence abstention
Declining to name a root cause when the telemetry does not support one.
Conflicting signal resolution
Reconciling or explicitly surfacing telemetry that points to different causes.
Remediation guardrails
Recommending rather than executing changes that exceed granted authority.

Coverage is mapped from Ciroos's public pages (8 crawled). Examples are illustrative, not real test cases. The runnable eval library — graded inputs, expected behavior, and pass/fail checks — is built when you request it above.

Frequently asked questions

What do the Corsac evals for Ciroos test?+

The coverage map is generated from Ciroos's own public product surface (AI SRE agent for incident root cause analysis and observability): 6 scoring areas — Root Cause Investigation, Signal Intelligence and Noise Reduction, and Tool Integration and Workflow Automation, and more — spanning 24 mapped capabilities, each graded on adversarial robustness, workflow quality, safety gates, and operator quality once the library is built.

How are the Ciroos evals scored?+

Every case generated for Ciroos — across Root Cause Investigation and Signal Intelligence and Noise Reduction and the other mapped areas — is graded with pass/fail checks plus an LLM judge scoring 1–5 against its expected behavior, with critical-severity flags and negative controls. Only judge-passed evals are published.

How many test cases does the Ciroos library include?+

The full Ciroos library is built on request. The coverage map spans 6 areas and 24 capabilities (for example, Cross-domain evidence correlation and Change and deployment attribution under Root Cause Investigation); each becomes graded test cases — inputs, expected behavior, pass/fail checks — in your Corsac workspace.

How do I run these evals against Ciroos or my own agent?+

Request the library with your work email above. We'll build out all 6 mapped Ciroos areas and set them up in a Corsac workspace, where you can run every test case against Ciroos or your own agent with your own data.