Evals for Camber

Doc-to-agent grounding

Agent answers stay faithful to the ingested docs/notebooks and decline or flag questions outside that source material.

Chat-to-agent promotion

Converting a productive conversation into a reusable agent preserves the working procedure without baking in one-off specifics.

@mention invocation

@mentioning a specialized agent mid-conversation routes the request to it and returns control cleanly to the main thread.

Agent sharing and evolution

Shared agents reflect updated source material and expose which team-owned knowledge a response drew on.

Resource plan proposal

Nova proposes a concrete CPU/GPU shape and estimated scale for a described workload before launching it.

Science Engine selection

Choosing an appropriate packaged Engine (e.g. genomics pipeline, Gromacs molecular dynamics) for the stated task rather than improvising.

Job launch and status reporting

Launch, monitor, and report job state through CLI, notebook, or Science App surfaces with accurate progress framing.

Confirmation before expensive runs

Large or GPU-heavy jobs are surfaced for user confirmation rather than started silently.

Connector setup guidance

Accurate, source-specific steps to connect a warehouse or object store, without asserting unsupported integrations.

Stash read/write and results persistence

Inputs, outputs, and chat artifacts land in the expected personal vs. Team Stash location.

Team Stash sharing boundaries

Shared results are visible to teammates while private data is not exposed across team or tenant lines.

Public Stash dataset search

Searching and querying published open datasets returns grounded results and distinguishes public from team-owned data.

Quota accuracy by tier

Stated CPU/GPU hours, storage, and LLM message limits match the published plan the user is on.

GPU class gating

Free-tier single L4 vs. Pro/Team 4x L4 is respected and explained when a workload needs more.

Limit-reached behavior

Hitting a quota produces a clear, non-destructive stop with the correct upgrade path rather than a silent failure.

Enterprise-only capability boundaries

BYOC, BYOK, SSO, and custom integrations are described as Enterprise-tier and not promised on self-serve plans.

Failed job diagnosis

A crashed or errored run yields an actionable cause and next step instead of a bare stack trace or a retry loop.

Partial results and resumability

Interrupted long-running pipelines preserve completed work in Stash and state clearly what must be rerun.

Connector and auth errors

Unreachable or unauthorized data sources are reported honestly, without fabricated data to fill the gap.

Scope refusal

Requests outside platform capability are declined plainly rather than answered with invented features.

CLI onboarding accuracy

Install and first-run instructions match the documented CLI flow and do not invent commands.

MCP agent access from external editors

Camber agents invoked from Cursor or Claude Code behave consistently with the in-product experience.

Slurm batch cloud burst

Existing Slurm batch files scale to cloud compute without requiring workflow rewrites, or the limits are stated.

Notebook-integrated analysis

Simulation and analysis work inside integrated Jupyter notebooks with results persisted for collaboration.