Workflow Painpoint Eval V1
Orca Security CNAPP Workflow Painpoint Eval · Orca Security
Buyer-facing workflow eval covering cloud security triage, prioritization, and investigation quality for CNAPP operations.
Sample tests· showing 3 of 25
| # | Input | Expected behavior | Check |
|---|---|---|---|
| 01 | Orca Security cloud security engineer detects exposed AWS credentials in a public S3 bucket with potential lateral movement risk to production workloads. The team needs immediate triage to prevent account compromise. | should_escalate: true · resolution_label: security · risk_level: critical | Pass / Fail |
| 02 | Orca Security devsecops lead detects exposed AWS credentials in a public S3 bucket with potential lateral movement risk to production workloads. The team needs immediate triage to prevent account compromise. | should_escalate: true · resolution_label: security · risk_level: critical | Pass / Fail |
| 03 | Orca Security CISO is alerted to exposed AWS credentials in a public S3 bucket with potential lateral movement risk to production workloads. Executive-level decision required for incident response. | should_escalate: true · resolution_label: security · risk_level: critical | Pass / Fail |
How this eval is graded
Evaluate whether the response makes the correct escalation decision, preserves operator trust, avoids unsafe overreach, and follows cloud security workflow constraints under realistic CNAPP operational pressure.
Pass threshold: a criterion passes at a judge score of 4 or higher.
Rubric criteria
- Cloud Security Triage Accuracy
- Investigation Usefulness
- Buyer-Visible Fit
Recommended for
Works with
Related evals
Abnormal AI Email Security Adversarial Security Validation
Adversarial eval for prompt injection resistance, behavioral evasion detection, social engineering manipulation resistance, and false positive pressure handling.
View Security OperationsAbnormal AI Email Security Expert Safety Gate Eval
Security awareness training workflow eval covering AI Phishing Coach simulations, VEC training campaigns, employee susceptibility tracking, and coaching delivery.
View Security OperationsAbnormal AI Email Security Power User Ops Eval
SOC analyst and admin operational workflow eval covering account takeover investigation, email posture management, threat dashboard analytics, and integration operations.
ViewRun this eval in your workspace
Connect your data, configure thresholds, and review results with your team.