Auth Workspaces Rbac Governance
LangSmith · LangSmith
LLM Observability & Evaluation Platform — LangSmith (LangChain)
LangSmith evals — Auth, Workspaces, RBAC & Governance (relift v3 InfraRed)
About LangSmith
LangSmith is LangChain's LLM observability and evaluation platform: tracing, datasets, evaluators (LLM-as-judge, code, and human), experiments, prompt management, and online monitoring used by AI teams to measure and improve LLM apps in production.
Employees
~200
Industry
LLM Observability
Headquarters
San Francisco, CA
Website
www.langchain.com/langsmithSample tests· showing 3 of 10
| # | Input | Expected behavior | Check |
|---|---|---|---|
| 01 | Operator generates a LANGSMITH_API_KEY from Settings → API Keys for a CI pipeline. CI must only see one workspace's data. | Create the key inside the intended workspace; the key is scoped to that workspace by construction and cannot read another workspace's projects or datasets. Store as a CI secret. Rotate on a schedule and on personnel change. Audit via Settings → API Keys → last-used. | Pass / FailAi Platformcritical |
| 02 | Engineering can write traces, finance can only read cost dashboards, and external auditors get read-only access to one project. | Use workspace roles: developer (read/write) for engineering, viewer (read) for finance, and an auditor role scoped to a single project for external auditors. Confirm RBAC scope via Settings → Members. Avoid granting Admin broadly. | Pass / FailAi Platformhigh |
| 03 | Customer A and Customer B are different workspaces in the operator's org. A user with access only to A's workspace must not see B's projects. | Use distinct workspaces per customer (or use ABAC roles per project if the Enterprise plan supports it). Verify isolation by attempting to list B's projects with an A-scoped key — expect 403 / empty list. Document the isolation model in the SOC 2 narrative. | Pass / FailAi Platformcritical |
How this eval is graded
Grade against expected.ideal_behavior and expected.rubric. Per-criterion pass requires mean >= 4.0 and no criterion below 3.
Rubric criteria
- Langsmith
- Ai Platform
- Auth Workspaces Rbac Governance
Recommended for
Works with
Related evals
Run this eval in your workspace
Connect your data, configure thresholds, and review results with your team.