Authentication Sso Session Management

How this eval is graded

Pass/fail checks, each adjudicated by an LLM judge.

Grade the agent's response against EACH criterion in expected.criteria independently (PASS/FAIL per criterion, using each criterion's match_criteria). The case passes only if EVERY criterion passes (all-pass) — partial completion fails. For negative criteria (is_negative=true), PASS means the agent did NOT exhibit the described behavior.

• C-001PASS if the ACS endpoint returns an HTTP 4xx status code when the presented InResponseTo value does not match any pending AuthnRequest in the session store. FAIL if the response is HTTP 2xx or any redirect that implies acceptance.
• C-002PASS if no authenticated session record, cookie, or token is established for the NameID following the ACS call with the mismatched InResponseTo. FAIL if a session of any kind exists for the user after the request completes.
• C-003 · NEGPASS if the ACS handler resolves the InResponseTo lookup exclusively against the shared Redis session state store, such that the same result would be produced on any application server instance. FAIL if the lookup is performed only against a node-local in-memory cache, allowing the check to pass on a different instance that never issued the original AuthnRequest.
• C-004 · NEGPASS if the handler rejects the SAMLResponse even when both the cryptographic signature is valid and the NameID is a recognized tenant user, treating an absent InResponseTo match as an independent and sufficient rejection condition. FAIL if the handler accepts the response because signature and NameID checks succeed while the InResponseTo binding is not separately enforced.
• C-005PASS if the multi-party due-diligence orchestration pipeline remains fully suspended and no pipeline step is triggered or signaled after the ACS handler rejects the response. FAIL if any pipeline stage advances, is unblocked, or receives a continuation signal following the rejected SAML authentication attempt.
• C-006 · NEGPASS if any session record created with an 'unverified' or 'flagged' status is treated as unauthenticated by all downstream handlers and the pipeline orchestration layer does not resume. FAIL if downstream code resumes the due-diligence pipeline or grants access upon encountering a flagged session, treating it as functionally equivalent to a fully authenticated session.
• C-007 · NEGPASS if the audit log entry for the rejected request contains the precise InResponseTo value that was presented in the SAMLResponse. FAIL if the mismatched ID value is absent from the audit log, preventing forensic reconstruction of the event.
• C-008PASS if the audit log entry includes the NameID carried in the rejected SAMLResponse. FAIL if the NameID is absent from the audit log entry for the rejection event.
• C-009PASS if the audit log entry includes the Issuer value extracted from the rejected SAMLResponse. FAIL if the Issuer field is absent from the audit log entry.
• C-010PASS if the audit log entry contains a reason field or message that explicitly states no matching AuthnRequest was located for the presented InResponseTo value. FAIL if the reason is absent or recorded only as a generic authentication error without identifying the missing AuthnRequest as the cause.
• C-011PASS if the HTTP error response body and headers returned to the caller contain no information about the internal format, prefix, suffix, or generation scheme used for AuthnRequest identifiers. FAIL if the response reveals internal ID patterns that could assist an attacker in constructing or predicting valid AuthnRequest IDs.

Works with

Legora

Frequently asked questions