Credentials And Auth
n8n (self-host + Cloud) · n8n
Workflow Automation — n8n
Evaluates n8n's Credentials & Auth across 9 scenario-based test cases, each graded against an expected-behavior rubric by an LLM judge, from Corsac's Workflow Automation eval coverage.
About n8n
n8n is an open-source workflow automation platform — visually composed workflows of 1000+ nodes including AI/LangChain nodes (AI Agent, vector stores, memory, tools), with triggers (webhook/schedule/poll/form/chat), credentials with encryption at rest, queue-mode execution (Redis-backed workers), self-host (Docker/Kubernetes) and n8n Cloud options, and source-control/embed for teams.
Sample tests· showing 3 of 9
| # | Input | Expected behavior | Check |
|---|---|---|---|
| 01 | Ops team rotates N8N_ENCRYPTION_KEY in the docker-compose env without re-encrypting existing credentials. All workflows fail with 'cannot decrypt credential'. | Rotation is a two-step playbook: run `n8n executeBatch --re-encrypt` (or the documented re-encryption procedure) with both old and new keys configured BEFORE removing the old key. Test on a staging instance. Back up the DB first. | Pass / FailAi Platformcritical |
| 02 | Operator registers an OAuth2 app at the vendor with redirect https://localhost:5678/rest/oauth2-credential/callback for dev, then ships to prod at https://n8n.acme.com without updating the vendor. | Register the production callback URL (https://<instance>/rest/oauth2-credential/callback) at the vendor before promoting. n8n shows the exact URL to register in the credential UI. Keep dev + prod as separate OAuth apps so rotating one doesn't bleed into the other. | Pass / FailAi Platformhigh |
| 03 | Operator requests `https://www.googleapis.com/auth/drive` (full Drive access) when the workflow only needs to download one shared folder. | Pick the narrowest scope that satisfies the use case (`drive.readonly` or `drive.file`). Document scopes per workflow. Re-consent users when scopes change — silent scope expansion is a breach of trust. | Pass / FailAi Platformhigh |
How this eval is graded
Grade against expected.ideal_behavior and expected.rubric. Per-criterion pass requires mean >= 4.0 and no criterion below 3.
Rubric criteria
- N8n
- Ai Platform
- Credentials And Auth
Recommended for
Works with
Related evals
Claude API
Evaluates Anthropic's Batch API across 9 scenario-based test cases, each graded against an expected-behavior rubric by an LLM judge, from Corsac's Foundation Model & API eval coverage.
View AI PlatformClaude API
Evaluates Anthropic's Extended Thinking across 9 scenario-based test cases, each graded against an expected-behavior rubric by an LLM judge, from Corsac's Foundation Model & API eval coverage.
View AI PlatformClaude API
Evaluates Anthropic's Files API & Citations across 9 scenario-based test cases, each graded against an expected-behavior rubric by an LLM judge, from Corsac's Foundation Model & API eval coverage.
ViewFrequently asked questions
What does the Credentials And Auth eval for n8n n8n (self-host + Cloud) test?+
Evaluates n8n's Credentials & Auth across 9 scenario-based test cases, each graded against an expected-behavior rubric by an LLM judge, from Corsac's Workflow Automation eval coverage.
How is the Credentials And Auth eval scored?+
The judge rubric: Grade against expected.ideal_behavior and expected.rubric. Per-criterion pass requires mean >= 4.0 and no criterion below 3.
How many test cases does this eval pack include?+
The Credentials And Auth pack for n8n n8n (self-host + Cloud) contains 9 test cases. 3 sample cases are shown free on this page; the full set runs in a Corsac workspace.
How do I run this eval?+
Sign up for Corsac, connect your model or agent endpoint, and run the Credentials And Auth pack as-is or after customizing thresholds. Results land in your workspace with per-case scores, and you can gate releases on the pack in CI via the REST API.
Run this eval in your workspace
Connect your data, configure thresholds, and review results with your team.