Workflows As Code And Api
n8n (self-host + Cloud) · n8n
Workflow Automation — n8n
Evaluates n8n's Workflows-as-Code & API across 9 scenario-based test cases, each graded against an expected-behavior rubric by an LLM judge, from Corsac's Workflow Automation eval coverage.
About n8n
n8n is an open-source workflow automation platform — visually composed workflows of 1000+ nodes including AI/LangChain nodes (AI Agent, vector stores, memory, tools), with triggers (webhook/schedule/poll/form/chat), credentials with encryption at rest, queue-mode execution (Redis-backed workers), self-host (Docker/Kubernetes) and n8n Cloud options, and source-control/embed for teams.
Sample tests· showing 3 of 9
| # | Input | Expected behavior | Check |
|---|---|---|---|
| 01 | Operator scripts a CI job that creates a workflow via POST /api/v1/workflows and supplies the API key in the Authorization header (Bearer). | Per docs, the public API authenticates via the X-N8N-API-KEY header, not Authorization: Bearer. Update the CI job to send X-N8N-API-KEY. Store the key in the CI secret store; rotate quarterly. Use a dedicated automation user. | Pass / FailAi Platformhigh |
| 02 | Operator runs `n8n export:workflow --all --output=./workflows.json` on prod and `n8n import:workflow --input=./workflows.json` on staging. Staging credentials fail. | Export contains workflow JSON + credential references by id, but NOT secret values (unless explicitly requested with --decrypted flag). Re-create credentials on staging with stage-specific secrets. Use source control for the workflow JSON to track diffs. [REQUIRES-VERIFICATION] for exact --decrypte… | Pass / FailAi Platformhigh |
| 03 | Operator uses Source Control (Enterprise) with dev/staging/prod environments. An untested workflow change is pushed straight to prod. | Configure dev as 'push' and prod as 'pull' (or follow the documented promote-via-PR pattern). Require PR review on the git remote for prod-bound changes. Tag releases. Test pull on staging before pulling on prod. | Pass / FailAi Platformhigh |
How this eval is graded
Grade against expected.ideal_behavior and expected.rubric. Per-criterion pass requires mean >= 4.0 and no criterion below 3.
Rubric criteria
- N8n
- Ai Platform
- Workflows As Code And Api
Recommended for
Works with
Related evals
Claude API
Evaluates Anthropic's Batch API across 9 scenario-based test cases, each graded against an expected-behavior rubric by an LLM judge, from Corsac's Foundation Model & API eval coverage.
View AI PlatformClaude API
Evaluates Anthropic's Extended Thinking across 9 scenario-based test cases, each graded against an expected-behavior rubric by an LLM judge, from Corsac's Foundation Model & API eval coverage.
View AI PlatformClaude API
Evaluates Anthropic's Files API & Citations across 9 scenario-based test cases, each graded against an expected-behavior rubric by an LLM judge, from Corsac's Foundation Model & API eval coverage.
ViewFrequently asked questions
What does the Workflows As Code And Api eval for n8n n8n (self-host + Cloud) test?+
Evaluates n8n's Workflows-as-Code & API across 9 scenario-based test cases, each graded against an expected-behavior rubric by an LLM judge, from Corsac's Workflow Automation eval coverage.
How is the Workflows As Code And Api eval scored?+
The judge rubric: Grade against expected.ideal_behavior and expected.rubric. Per-criterion pass requires mean >= 4.0 and no criterion below 3.
How many test cases does this eval pack include?+
The Workflows As Code And Api pack for n8n n8n (self-host + Cloud) contains 9 test cases. 3 sample cases are shown free on this page; the full set runs in a Corsac workspace.
How do I run this eval?+
Sign up for Corsac, connect your model or agent endpoint, and run the Workflows As Code And Api pack as-is or after customizing thresholds. Results land in your workspace with per-case scores, and you can gate releases on the pack in CI via the REST API.
Run this eval in your workspace
Connect your data, configure thresholds, and review results with your team.