Auth Workspaces Rbac Governance
LangSmith · LangSmith
LLM Observability & Evaluation Platform — LangSmith (LangChain)
Evaluates LangSmith's Auth, Workspaces, RBAC & Governance across 10 scenario-based test cases, each graded against an expected-behavior rubric by an LLM judge, from Corsac's LLM Observability & Evaluation Platform eval coverage.
About LangSmith
LangSmith is LangChain's LLM observability and evaluation platform: tracing, datasets, evaluators (LLM-as-judge, code, and human), experiments, prompt management, and online monitoring used by AI teams to measure and improve LLM apps in production.
Employees
~200
Industry
LLM Observability
Headquarters
San Francisco, CA
Website
www.langchain.com/langsmithSample tests· showing 3 of 10
| # | Input | Expected behavior | Check |
|---|---|---|---|
| 01 | Operator generates a LANGSMITH_API_KEY from Settings → API Keys for a CI pipeline. CI must only see one workspace's data. | Create the key inside the intended workspace; the key is scoped to that workspace by construction and cannot read another workspace's projects or datasets. Store as a CI secret. Rotate on a schedule and on personnel change. Audit via Settings → API Keys → last-used. | Pass / FailAi Platformcritical |
| 02 | Engineering can write traces, finance can only read cost dashboards, and external auditors get read-only access to one project. | Use workspace roles: developer (read/write) for engineering, viewer (read) for finance, and an auditor role scoped to a single project for external auditors. Confirm RBAC scope via Settings → Members. Avoid granting Admin broadly. | Pass / FailAi Platformhigh |
| 03 | Customer A and Customer B are different workspaces in the operator's org. A user with access only to A's workspace must not see B's projects. | Use distinct workspaces per customer (or use ABAC roles per project if the Enterprise plan supports it). Verify isolation by attempting to list B's projects with an A-scoped key — expect 403 / empty list. Document the isolation model in the SOC 2 narrative. | Pass / FailAi Platformcritical |
How this eval is graded
Grade against expected.ideal_behavior and expected.rubric. Per-criterion pass requires mean >= 4.0 and no criterion below 3.
Rubric criteria
- Langsmith
- Ai Platform
- Auth Workspaces Rbac Governance
Recommended for
Works with
Related evals
Claude API
Evaluates Anthropic's Batch API across 9 scenario-based test cases, each graded against an expected-behavior rubric by an LLM judge, from Corsac's Foundation Model & API eval coverage.
View AI PlatformClaude API
Evaluates Anthropic's Extended Thinking across 9 scenario-based test cases, each graded against an expected-behavior rubric by an LLM judge, from Corsac's Foundation Model & API eval coverage.
View AI PlatformClaude API
Evaluates Anthropic's Files API & Citations across 9 scenario-based test cases, each graded against an expected-behavior rubric by an LLM judge, from Corsac's Foundation Model & API eval coverage.
ViewFrequently asked questions
What does the Auth Workspaces Rbac Governance eval for LangSmith LangSmith test?+
Evaluates LangSmith's Auth, Workspaces, RBAC & Governance across 10 scenario-based test cases, each graded against an expected-behavior rubric by an LLM judge, from Corsac's LLM Observability & Evaluation Platform eval coverage.
How is the Auth Workspaces Rbac Governance eval scored?+
The judge rubric: Grade against expected.ideal_behavior and expected.rubric. Per-criterion pass requires mean >= 4.0 and no criterion below 3.
How many test cases does this eval pack include?+
The Auth Workspaces Rbac Governance pack for LangSmith LangSmith contains 10 test cases. 3 sample cases are shown free on this page; the full set runs in a Corsac workspace.
How do I run this eval?+
Sign up for Corsac, connect your model or agent endpoint, and run the Auth Workspaces Rbac Governance pack as-is or after customizing thresholds. Results land in your workspace with per-case scores, and you can gate releases on the pack in CI via the REST API.
Run this eval in your workspace
Connect your data, configure thresholds, and review results with your team.