Evals for SkyPilot

Backend and cloud coverage

Which execution backends are supported — Kubernetes, Slurm, VMs, hyperscalers, and neoclouds — and how a workload targets them.

Single-interface launch

Expressing a workload once (YAML/CLI) and running it unchanged across different backends.

Migration off Slurm

Moving existing Slurm-based training and cluster workflows onto SkyPilot, as described in the Abridge, H Company, and Hippocratic AI stories.

Heterogeneous accelerator fleets

Operating across mixed accelerator generations and SKUs (e.g. H100 and H200 in one pool) rather than a single homogeneous cluster.

Multi-node and gang scheduling

Coordinated placement of multi-node jobs across a cluster, called out as an out-of-the-box capability by Applied Compute.

Interactive development access

SSH and cluster access for iterating on jobs, not just batch submission.

Large-fleet operation

Managing workloads on very large existing fleets, e.g. Meta FAIR's 100k+ GPU Slurm environment.

Storage and data path for training

Object store configuration and tuning for distributed training, per the object-store benchmarking work.

Independent inference scaling

Job Groups that let rollout/inference capacity scale separately from training in an RL loop.

RL on Kubernetes at scale

Scaling online RL to thousands of GPUs on Kubernetes, as in H Company's 2,000+ GPU deployment.

Limits of batch scheduling for RL

Explaining why Slurm's batch model conflicts with online RL and what changes on SkyPilot.

RL framework integration

Running established RL stacks (e.g. slime) on top of SkyPilot's job primitives.

Endpoint definition from one YAML

Declaring and launching a production endpoint on an existing cluster with a single configuration.

Large open-weight model serving

Serving multi-hundred-billion to trillion-parameter open-weight models across owned GPU clusters, e.g. Kimi K3 at 2.8T parameters.

Endpoint operations

Running and managing inference endpoints on every cluster a team owns, across backends.

Untrusted code isolation

Executing agent-generated code safely on Kubernetes clusters the customer already operates.

Sandbox scale and launch latency

Published characteristics: 50,000+ sandboxes sustained on a single cluster, individual launches under a second.

Multi-cluster sandbox fanout

Extending sandbox capacity beyond one cluster when a single cluster's ceiling is reached.

Data residency and cost posture

Code and data never leaving the customer's cloud, at up to one-tenth the cost of hosted alternatives.

GPU Compass offering discovery

Navigating 2,000+ GPU offerings across 20+ clouds to compare available hardware.

Capacity shortfall handling

Behavior when a preferred provider has no capacity, including sourcing across reservations and bursty resources.

Cost-aware hardware selection

Choosing between accelerator tiers for a given stage of work, such as screening on cheaper GPUs and validating on faster ones.