Eval directory · AI Platform

Evals for Restate

Evaluation packs covering adversarial robustness, safety gates, workflow quality, and operator-level checks for Restate AI products.

About Restate

Restate is a durable-execution runtime that makes AI agents, workflows, and backend services resilient to failures, with persistent progress and reliable communication.

Industry

Durable Execution Infrastructure

Website

www.restate.dev

Use the eval library for Restate

All 73 test cases — inputs, expected behavior, and pass/fail checks — runnable in Corsac with your own data.

Related in AI Platform

All evals →

More AI Platform eval libraries

Coverage map

What would you measure for Restate?

8 areas · 73 graded scenarios

Every eval set is graded on

Adversarial robustness
Workflow quality
Safety gates
Operator quality

Pass/Fail + LLM judge 1–5 · critical severity flags · negative controls

Durable Handlers And Journaling

Evaluates Restate's Durable Handlers & Journaled Execution across 9 scenario-based test cases, each graded against an expected-behavior rubric by an LLM judge, from Corsac's Durable Execution eval coverage.

Mapped capabilities

9 scenarios

ctx.run wraps side effects
determinism on replay
retries are not idempotent in the side effect

Public sample case

Input: A Restate durable handler charges a card by calling Stripe inside the handler body directly (a bare `await stripe.charge(...)`), not wrapped in a durable step. The process crashes after the charge but before the handler's next li…
Expected behavior: Wrap every non-deterministic side effect (external HTTP call, random, time, UUID) in ctx.run(...) so its result is journaled. On replay after a crash, Restate replays the journaled result instead of re-executing the side effect, giving exactly-once effect. A bare call outside ctx.run re-executes on…
Check: Pass / fail check

Durable Promises And Awakeables

Evaluates Restate's Durable Promises & Awakeables across 9 scenario-based test cases, each graded against an expected-behavior rubric by an LLM judge, from Corsac's Durable Execution eval coverage.

Mapped capabilities

9 scenarios

awakeable id handoff to external system
resolve vs reject awakeable
awakeable timeout / abandonment

Public sample case

Input: A handler creates an awakeable and must hand its id to a third-party callback so the third party can resolve it later. The handler stores the id only in a local variable and awaits.
Expected behavior: Create the awakeable with ctx.awakeable(), then durably hand its id to the external system inside a journaled step (the call that registers the callback) BEFORE awaiting it — so a crash between creation and handoff does not orphan a promise nobody can resolve. The id is the rendezvous token; treat …
Check: Pass / fail check

Durable Timers And Scheduling

Evaluates Restate's Durable Timers & Scheduling across 9 scenario-based test cases, each graded against an expected-behavior rubric by an LLM judge, from Corsac's Durable Execution eval coverage.

Mapped capabilities

9 scenarios

ctx.sleep durable delay
delayed invocation / send with delay
cron / recurring schedule

Public sample case

Input: A reminder handler must fire 24 hours after an event. The integrator schedules an OS cron or in-process setTimeout to call back.
Expected behavior: Use ctx.sleep(24h) inside the durable handler: the invocation suspends and Restate resumes it via a durable timer after the delay, surviving restarts and redeploys with no external scheduler. An in-process timer dies with the process; a cron is a separate system to keep consistent. Keep the delay d…
Check: Pass / fail check

Observability Security And Ops

Evaluates Restate's Observability, Security & Operations across 10 scenario-based test cases, each graded against an expected-behavior rubric by an LLM judge, from Corsac's Durable Execution eval coverage.

Mapped capabilities

10 scenarios

introspect a stuck invocation
distributed tracing across handlers
ingress authentication

Server Deployment And Registration

Evaluates Restate's Restate Server, Deployment & Registration across 9 scenario-based test cases, each graded against an expected-behavior rubric by an LLM judge, from Corsac's Durable Execution eval coverage.

Mapped capabilities

9 scenarios

register a service deployment
immutable deployment versioning
self-hosted server state durability

Service Invocation And Idempotency

Evaluates Restate's Service Invocation & Idempotency across 9 scenario-based test cases, each graded against an expected-behavior rubric by an LLM judge, from Corsac's Durable Execution eval coverage.

Mapped capabilities

9 scenarios

request/response ctx.call
one-way send (fire-and-forget)
idempotency key on ingress invocation

Virtual Objects And State

Evaluates Restate's Virtual Objects & State across 9 scenario-based test cases, each graded against an expected-behavior rubric by an LLM judge, from Corsac's Durable Execution eval coverage.

Mapped capabilities

9 scenarios

single-writer per object key
shared (read-only) handlers concurrency
ctx.set / ctx.get K/V state

Workflows And Sagas

Evaluates Restate's Workflows & Sagas across 9 scenario-based test cases, each graded against an expected-behavior rubric by an LLM judge, from Corsac's Durable Execution eval coverage.

Mapped capabilities

9 scenarios

workflow run-once semantics
saga compensation on step failure
workflow signals from outside

Frequently asked questions

What do the Corsac evals for Restate test?+

Each eval pack tests Restate's public product surface — including Durable Handlers And Journaling, Durable Promises And Awakeables, and Durable Timers And Scheduling — against graded scenarios covering adversarial robustness, workflow quality, safety gates, and operator quality. Every pack is runnable in Corsac with your own data.

How are the Restate evals scored?+

Pass/fail checks plus an LLM judge scoring 1–5 against each of the 73 Restate cases — from Observability Security And Ops (10 scenarios) down to the smallest pack — its own expected behavior, with critical-severity flags and negative controls. Only judge-passed evals are published to the Restate library.

How many test cases does the Restate library include?+

The Restate eval library includes 73 graded test cases across 8 eval packs, the largest being Observability Security And Ops with 10 scenarios. Each case defines an input, expected behavior, and pass/fail criteria.

How do I run these evals against Restate or my own agent?+

Request the library with your work email above and we'll set it up in a Corsac workspace, where you can run all 8 Restate packs — Durable Handlers And Journaling and Durable Promises And Awakeables and the rest — against Restate or your own agent with your own data.