Online Monitoring Alerting
LangSmith · LangSmith
LLM observability and evaluation — LangSmith
LangSmith evals — Online Monitoring & Alerting (relift v3)
About LangSmith
LangSmith is LangChain's LLM observability and evaluation platform: tracing, datasets, evaluators (LLM-as-judge, code, and human), experiments, prompt management, and online monitoring used by AI teams to measure and improve LLM apps in production.
Employees
~200
Industry
LLM Observability
Headquarters
San Francisco, CA
Website
www.langchain.com/langsmithSample tests· showing 3 of 8
| # | Input | Expected behavior | Check |
|---|---|---|---|
| 01 | Production project 'support-bot' sees tool exceptions; on-call uses LangSmith Alerts UI. | Configure alert on error percentage threshold with filter scoped to project; include webhook notification; test alert preview before enable. | Pass / FailAi Platformcritical |
| 02 | Runs tagged env=prod; SRE wants latency threshold not error count. | Use latency metric in alert builder with tag filter env=prod; set window and threshold; link to run query from alert payload. | Pass / FailAi Platformhigh |
| 03 | Custom receiver builds deep links into LangSmith UI for on-call. | Use payload fields for project, run ids, alert rule name, metric value; construct UI URL from documented host pattern; avoid logging full PII payloads. | Pass / FailAi Platformmedium |
How this eval is graded
Grade against expected.ideal_behavior and expected.rubric. Penalize failure_modes.
Rubric criteria
- Langsmith
- Ai Platform
- Online Monitoring Alerting
Recommended for
Works with
Related evals
Run this eval in your workspace
Connect your data, configure thresholds, and review results with your team.