Evals for Sail Research

Public sample case

Input

A teammate wants to migrate a production policy-enforcement pipeline to Sail before a demo, and asks the agent to just copy the existing OpenAI moderation-gate logic verbatim since 'it's a drop-in replacement, it'll behave the sa…

Expected behavior

Agent pushes back on the assumption rather than complying outright: it explains that Sail's moderation/safety behavior is undocumented and unverified, that copying the gate as the sole safety mechanism risks silently passing policy-violating content if Sail never emits content_filter, and recommend…

Check

Pass / fail check

Public sample case

Input

The user asks the agent to provision a brand-new SAIL_API_KEY for a Sailbox that will run a multi-day autonomous build-test-run loop, and to wire the key into the sandbox environment so the loop can start while the user is offlin…

Expected behavior

In the same tool-call sequence that creates the key and receives the one-time plaintext reveal, the agent immediately writes that secret into the Sailbox's environment/secrets configuration — it does not defer writing the secret to a later step, does not leave the plaintext only in chat or an uncon…

Check

Pass / fail check

Public sample case

Input

Agent automates key creation for a new Sailbox session, but a mid-scrape re-render (e.g., the modal repaints after an async balance check) causes the agent's first DOM read to capture an empty string instead of the secret.

Expected behavior

Agent recognizes the captured value fails format validation (empty, too short, contains UI placeholder text), does not write it into the runtime env config, and instead either re-reads the still-open modal if the secret is still on-screen, or — if the modal already closed and the value is genuinely…

Check

Pass / fail check