Execution Engine Queue Mode And Scaling
n8n (self-host + Cloud) · n8n
Workflow Automation — n8n
Evaluates n8n's Execution Engine, Queue Mode & Scaling across 9 scenario-based test cases, each graded against an expected-behavior rubric by an LLM judge, from Corsac's Workflow Automation eval coverage.
About n8n
n8n is an open-source workflow automation platform — visually composed workflows of 1000+ nodes including AI/LangChain nodes (AI Agent, vector stores, memory, tools), with triggers (webhook/schedule/poll/form/chat), credentials with encryption at rest, queue-mode execution (Redis-backed workers), self-host (Docker/Kubernetes) and n8n Cloud options, and source-control/embed for teams.
Sample tests· showing 3 of 9
| # | Input | Expected behavior | Check |
|---|---|---|---|
| 01 | Operator runs n8n in a single Docker container under load. Workflow latency spikes because the main process is both serving the UI and running long workflows. | Switch to EXECUTIONS_MODE=queue with a Redis URL and one or more `n8n worker` processes. Main process handles UI + trigger registration; workers consume jobs. Pin worker concurrency via N8N_CONCURRENCY_PRODUCTION_LIMIT per worker. Do not leave both modes mixed across instances. | Pass / FailAi Platformcritical |
| 02 | A workflow includes a Code node with an accidental infinite loop. The worker hangs indefinitely. | Set EXECUTIONS_TIMEOUT (seconds) at the instance level and EXECUTIONS_TIMEOUT_MAX as the operator-facing ceiling. Set per-workflow `executionTimeout` for long legitimate jobs. Combine with Code-node sandbox to abort runaway scripts. | Pass / FailAi Platformhigh |
| 03 | A queue-mode worker OOMs in the middle of a long HTTP node. Bull marks the job stalled. | n8n's behavior on stalled jobs depends on Bull's stalled-job recovery + the operator's QUEUE_RECOVERY_INTERVAL [REQUIRES-VERIFICATION]. Verify whether the workflow is retried from scratch, marked failed, or resumed at the node. Design workflows to be idempotent end-to-end (esp. write nodes) so retr… | Pass / FailAi Platformcritical |
How this eval is graded
Grade against expected.ideal_behavior and expected.rubric. Per-criterion pass requires mean >= 4.0 and no criterion below 3.
Rubric criteria
- N8n
- Ai Platform
- Execution Engine Queue Mode And Scaling
Recommended for
Works with
Related evals
Claude API
Evaluates Anthropic's Batch API across 9 scenario-based test cases, each graded against an expected-behavior rubric by an LLM judge, from Corsac's Foundation Model & API eval coverage.
View AI PlatformClaude API
Evaluates Anthropic's Extended Thinking across 9 scenario-based test cases, each graded against an expected-behavior rubric by an LLM judge, from Corsac's Foundation Model & API eval coverage.
View AI PlatformClaude API
Evaluates Anthropic's Files API & Citations across 9 scenario-based test cases, each graded against an expected-behavior rubric by an LLM judge, from Corsac's Foundation Model & API eval coverage.
ViewFrequently asked questions
What does the Execution Engine Queue Mode And Scaling eval for n8n n8n (self-host + Cloud) test?+
Evaluates n8n's Execution Engine, Queue Mode & Scaling across 9 scenario-based test cases, each graded against an expected-behavior rubric by an LLM judge, from Corsac's Workflow Automation eval coverage.
How is the Execution Engine Queue Mode And Scaling eval scored?+
The judge rubric: Grade against expected.ideal_behavior and expected.rubric. Per-criterion pass requires mean >= 4.0 and no criterion below 3.
How many test cases does this eval pack include?+
The Execution Engine Queue Mode And Scaling pack for n8n n8n (self-host + Cloud) contains 9 test cases. 3 sample cases are shown free on this page; the full set runs in a Corsac workspace.
How do I run this eval?+
Sign up for Corsac, connect your model or agent endpoint, and run the Execution Engine Queue Mode And Scaling pack as-is or after customizing thresholds. Results land in your workspace with per-case scores, and you can gate releases on the pack in CI via the REST API.
Run this eval in your workspace
Connect your data, configure thresholds, and review results with your team.