Eval directory · Code Assistant

Evals for Temporal

Evaluation packs covering adversarial robustness, safety gates, workflow quality, and operator-level checks for Temporal AI products.

About Temporal

Temporal is a durable execution platform for orchestrating long-running, fault-tolerant workflows. Code runs as deterministic workflows backed by an event-sourced history with automatic retries, timeouts, signals, and safe versioning.

Employees

~300

Industry

Developer Infrastructure

Headquarters

Seattle, WA

Website

temporal.io

Use the eval library for Temporal

All 48 test cases — inputs, expected behavior, and pass/fail checks — runnable in Corsac with your own data.

Related in Code Assistant

All evals →

More Code Assistant eval libraries

Coverage map

What would you measure for Temporal?

8 areas · 48 graded scenarios

Every eval set is graded on

Adversarial robustness
Workflow quality
Safety gates
Operator quality

Pass/Fail + LLM judge 1–5 · critical severity flags · negative controls

Activity Retries Timeouts

Evaluates Temporal's Activity Retries & Timeouts across 6 scenario-based test cases, each graded against an expected-behavior rubric by an LLM judge, from Corsac's Durable Execution & Workflow Orchestration eval coverage.

Mapped capabilities

6 scenarios

RetryPolicy backoff tuning
scheduleToClose vs startToClose
heartbeatTimeout and RecordHeartbeat

Public sample case

Input: ActivityOptions RetryPolicy maximumAttempts=0 with backoffCoefficient=2 initialInterval=1s calling Stripe; finance sees unbounded retries overnight.
Expected behavior: Agent sets maximumAttempts (e.g., 5-10), caps maximumInterval, marks 429 retryable only if idempotent POST with idempotency key; documents scheduleToClose bound.
Check: Pass / fail check

Child Workflows Continue As New

Evaluates Temporal's Child Workflows & Continue-As-New across 6 scenario-based test cases, each graded against an expected-behavior rubric by an LLM judge, from Corsac's Durable Execution & Workflow Orchestration eval coverage.

Mapped capabilities

6 scenarios

ParentClosePolicy ABANDON vs TERMINATE
child workflow id reuse
history bloat continue-as-new threshold

Public sample case

Input: Compliance child must finish logging when parent order cancelled; ParentClosePolicy TERMINATE kills child mid-write.
Expected behavior: Agent sets ParentClosePolicy ABANDON (or REQUEST_CANCEL per requirements), documents behavior in runbook, tests parent cancellation.
Check: Pass / fail check

Failure Handling Compensation

Evaluates Temporal's Failure Handling & Compensation across 6 scenario-based test cases, each graded against an expected-behavior rubric by an LLM judge, from Corsac's Durable Execution & Workflow Orchestration eval coverage.

Mapped capabilities

6 scenarios

saga compensation order
terminal vs retryable failure
Activity cancellation cleanup

Public sample case

Input: Saga books flight, hotel, payment; hotel fails; must compensate payment then flight in reverse order.
Expected behavior: Agent implements compensation stack (LIFO), each compensating Activity idempotent, workflow records completed forward steps.
Check: Pass / fail check

Signals Queries Updates

Evaluates Temporal's Signals, Queries & Updates across 6 scenario-based test cases, each graded against an expected-behavior rubric by an LLM judge, from Corsac's Durable Execution & Workflow Orchestration eval coverage.

Mapped capabilities

6 scenarios

Signal handler validation
Query read-only guarantee
Update validate-then-act

Task Queues Worker Scaling

Evaluates Temporal's Task Queues & Worker Scaling across 6 scenario-based test cases, each graded against an expected-behavior rubric by an LLM judge, from Corsac's Durable Execution & Workflow Orchestration eval coverage.

Mapped capabilities

6 scenarios

sticky queue affinity
poller starvation backpressure
separate workflow/activity queues

Versioning Patching

Evaluates Temporal's Versioning & Patching across 6 scenario-based test cases, each graded against an expected-behavior rubric by an LLM judge, from Corsac's Durable Execution & Workflow Orchestration eval coverage.

Mapped capabilities

6 scenarios

workflow.GetVersion branches
Worker Build ID deployment
patched() deprecation path

Visibility Search Attributes

Evaluates Temporal's Visibility & Search Attributes across 6 scenario-based test cases, each graded against an expected-behavior rubric by an LLM judge, from Corsac's Durable Execution & Workflow Orchestration eval coverage.

Mapped capabilities

6 scenarios

UpsertSearchAttributes typing
ListWorkflowExecutions filter syntax
indexing lag discovery gap

Workflow Determinism Replay

Evaluates Temporal's Workflow Determinism & Replay across 6 scenario-based test cases, each graded against an expected-behavior rubric by an LLM judge, from Corsac's Durable Execution & Workflow Orchestration eval coverage.

Mapped capabilities

6 scenarios

non-deterministic time detection
SideEffect for one-time external reads
replay-safe random and UUID

Frequently asked questions

What do the Corsac evals for Temporal test?+

Each eval pack tests Temporal's public product surface — including Activity Retries Timeouts, Child Workflows Continue As New, and Failure Handling Compensation — against graded scenarios covering adversarial robustness, workflow quality, safety gates, and operator quality. Every pack is runnable in Corsac with your own data.

How are the Temporal evals scored?+

Pass/fail checks plus an LLM judge scoring 1–5 against each of the 48 Temporal cases — from Activity Retries Timeouts (6 scenarios) down to the smallest pack — its own expected behavior, with critical-severity flags and negative controls. Only judge-passed evals are published to the Temporal library.

How many test cases does the Temporal library include?+

The Temporal eval library includes 48 graded test cases across 8 eval packs, the largest being Activity Retries Timeouts with 6 scenarios. Each case defines an input, expected behavior, and pass/fail criteria.

How do I run these evals against Temporal or my own agent?+

Request the library with your work email above and we'll set it up in a Corsac workspace, where you can run all 8 Temporal packs — Activity Retries Timeouts and Child Workflows Continue As New and the rest — against Temporal or your own agent with your own data.