Evals for Shortwave

Prompt-to-action fidelity

The executed mailbox operation matches the user's stated intent in verb, target set, and scope, without silently widening from a few threads to the whole inbox.

Destructive and bulk operation handling

Sweep, bulk triage, and archive-at-scale requests surface what will be affected and stop for confirmation rather than acting unilaterally on ambiguous scope.

Ambiguous or underspecified instructions

Requests that omit a time window, sender, or label resolve by asking a clarifying question instead of guessing a destructive default.

Todo and importance identification

Identifying important emails and todos in one pass, including behavior when nothing in the inbox qualifies.

Natural-language to filter semantics

A plain-English rule is translated into the intended matching condition and action, including negations, exclusions, and compound conditions.

Filter count limits by tier

Enforcement of the documented per-plan filter allowances and the behavior presented when a user tries to exceed them.

Auto-apply label behavior

Labels applied automatically behave consistently with the user's managed auto-apply settings and can be inspected or reversed.

Conflicting or overlapping rules

Two filters whose conditions overlap on the same thread resolve deterministically rather than producing order-dependent results.

Answer grounding and citation

Synthesized answers are traceable to specific retrieved threads or attachments rather than to model priors.

Search operator correctness

Documented search operators and combined terms return the threads the operator semantics specify, including substring matching behavior.

Retrieval limit transparency

When a query exceeds the plan's max-threads-per-search or search-history window, the response discloses that the result set was capped rather than presenting it as exhaustive.

Not-found and insufficient-evidence handling

Queries with no supporting email in the corpus return an explicit no-answer rather than a plausible fabrication.

Voice and style fidelity

Generated drafts match the register, length, and sign-off conventions of the user's sent mail.

Factual accuracy of injected content

Links, facts, and phrases surfaced by autocomplete correspond to real items in the user's email history and are not invented.

Instruction adherence in enhance/rewrite

Rewrite requests change what was asked and preserve the rest of the message, including quoted context and recipients.

Sensitive content restraint

Drafts avoid asserting commitments, figures, or confidential details the user did not supply or that are not present in the thread.

Availability and event creation accuracy

Proposed times reflect actual calendar availability, including time zone and existing-conflict handling.

Integration action scoping

Actions taken in connected tools stay within what the user requested and within the connection's granted permissions.

Connector failure and degradation

An unavailable, unauthorized, or timed-out integration is reported plainly instead of being reported as a completed action.

Tasklet hand-off boundaries

Work delegated to Tasklet for unattended automation is represented accurately, including that drafts and comments await human review.

Tier and quota enforcement

Intelligence level, daily AI usage, search history window, and thread caps behave according to the user's Business, Premier, or Max entitlement.

Multi-account and cross-mailbox isolation

With several accounts connected, AI actions and results stay scoped to the intended account and do not leak across mailboxes.

Team search access boundaries

Team-wide AI search surfaces only mail the requesting user is entitled to see.

Migration and history import integrity

Imported history from Gmail, Spark, or Superhuman is represented accurately, including partial or in-progress import state.