Workspaces Rbac Billing
LangSmith · LangSmith
LLM observability and evaluation — LangSmith
LangSmith evals — Workspaces, RBAC & Billing (relift v3)
About LangSmith
LangSmith is LangChain's LLM observability and evaluation platform: tracing, datasets, evaluators (LLM-as-judge, code, and human), experiments, prompt management, and online monitoring used by AI teams to measure and improve LLM apps in production.
Employees
~200
Industry
LLM Observability
Headquarters
San Francisco, CA
Website
www.langchain.com/langsmithSample tests· showing 3 of 9
| # | Input | Expected behavior | Check |
|---|---|---|---|
| 01 | Key must write traces/experiments but not delete datasets. | Create key in Settings → API Keys scoped to workspace; store in CI secret; rotate; never commit langsmith key to repo. | Pass / FailAi Platformcritical |
| 02 | Rotation scheduled; dual-key period needed for CI and prod workers. | Provision new key; dual-write period; update secrets; revoke old after drain; monitor 401 on ingestion. | Pass / FailAi Platformhigh |
| 03 | Procurement needs exact retention days for contract appendix. | State exact Plus plan retention days vary by marketing vs contract and are [REQUIRES-VERIFICATION]; point to FAQ/trust docs; do not invent 30/90 day numbers without source. | Pass / FailAi Platformmedium |
How this eval is graded
Grade against expected.ideal_behavior and expected.rubric. Penalize failure_modes.
Rubric criteria
- Langsmith
- Ai Platform
- Workspaces Rbac Billing
Recommended for
Works with
Related evals
Run this eval in your workspace
Connect your data, configure thresholds, and review results with your team.