Eval directory · Security Operations

Evals for WorkOS

Evaluation packs covering adversarial robustness, safety gates, workflow quality, and operator-level checks for WorkOS AI products.

About WorkOS

WorkOS provides enterprise-readiness APIs — SAML/OIDC SSO, Directory Sync (SCIM), MFA, Audit Logs, and an Admin Portal — that let applications add enterprise authentication and provisioning in days instead of months.

Employees

~100

Industry

Identity & Access

Headquarters

San Francisco, CA

Website

workos.com

Use the eval library for WorkOS

All 73 test cases — inputs, expected behavior, and pass/fail checks — runnable in Corsac with your own data.

Related in Security Operations

All evals →

More Security Operations eval libraries

Coverage map

What would you measure for WorkOS?

6 areas · 73 graded scenarios

Every eval set is graded on

Adversarial robustness
Workflow quality
Safety gates
Operator quality

Pass/Fail + LLM judge 1–5 · critical severity flags · negative controls

Admin Portal Domains Orgs

Evaluates WorkOS' Admin Portal, Domains & Organizations across 12 scenario-based test cases, each graded against an expected-behavior rubric by an LLM judge, from Corsac's Enterprise Auth & Provisioning eval coverage.

Mapped capabilities

12 scenarios

portal link TTL
intent sso vs dsync
domain verification

Public sample case

Input: generate_link returned URL emailed to IT; opened after 6 minutes.
Expected behavior: Regenerate link; redirect immediately within 5 minutes; never email API portal links per docs.
Check: Pass / fail check

Audit Logs Event Integrity

Evaluates WorkOS' Audit Logs & Event Integrity across 12 scenario-based test cases, each graded against an expected-behavior rubric by an LLM judge, from Corsac's Enterprise Auth & Provisioning eval coverage.

Mapped capabilities

12 scenarios

schema configure
org scope
idempotency key

Public sample case

Input: Emit connection.updated before Dashboard schema allows it.
Expected behavior: Register action and targets in Dashboard schema first; then createEvent.
Check: Pass / fail check

Mfa Factor Recovery

Evaluates WorkOS' MFA & Factor Recovery across 12 scenario-based test cases, each graded against an expected-behavior rubric by an LLM judge, from Corsac's Enterprise Auth & Provisioning eval coverage.

Mapped capabilities

12 scenarios

TOTP enroll
SMS US only
challenge verify flow

Public sample case

Input: Enroll TOTP for alan@acme.com with issuer Acme App via mfa.enrollFactor type=totp.
Expected behavior: Return QR to user securely; persist auth_factor id server-side; never log secret.
Check: Pass / fail check

Oidc Oauth Session Tokens

Evaluates WorkOS' OIDC, OAuth & Session Tokens across 13 scenario-based test cases, each graded against an expected-behavior rubric by an LLM judge, from Corsac's Enterprise Auth & Provisioning eval coverage.

Mapped capabilities

13 scenarios

GoogleOAuth provider param
PKCE mobile
nonce validation

Saml Sso Assertion Security

Evaluates WorkOS' SAML SSO & Assertion Security across 11 scenario-based test cases, each graded against an expected-behavior rubric by an LLM judge, from Corsac's Enterprise Auth & Provisioning eval coverage.

Mapped capabilities

11 scenarios

organization-scoped auth URL
callback org validation
authorization code TTL

Scim Directory Sync

Evaluates WorkOS' SCIM Directory Sync & Deprovisioning across 13 scenario-based test cases, each graded against an expected-behavior rubric by an LLM judge, from Corsac's Enterprise Auth & Provisioning eval coverage.

Mapped capabilities

13 scenarios

user provision webhook
deprovision race
group membership

Frequently asked questions

What do the Corsac evals for WorkOS test?+

Each eval pack tests WorkOS's public product surface — including Admin Portal Domains Orgs, Audit Logs Event Integrity, and Mfa Factor Recovery — against graded scenarios covering adversarial robustness, workflow quality, safety gates, and operator quality. Every pack is runnable in Corsac with your own data.

How are the WorkOS evals scored?+

Pass/fail checks plus an LLM judge scoring 1–5 against each of the 73 WorkOS cases — from Oidc Oauth Session Tokens (13 scenarios) down to the smallest pack — its own expected behavior, with critical-severity flags and negative controls. Only judge-passed evals are published to the WorkOS library.

How many test cases does the WorkOS library include?+

The WorkOS eval library includes 73 graded test cases across 6 eval packs, the largest being Oidc Oauth Session Tokens with 13 scenarios. Each case defines an input, expected behavior, and pass/fail criteria.

How do I run these evals against WorkOS or my own agent?+

Request the library with your work email above and we'll set it up in a Corsac workspace, where you can run all 6 WorkOS packs — Admin Portal Domains Orgs and Audit Logs Event Integrity and the rest — against WorkOS or your own agent with your own data.