Deployments

Operator runs a customer-facing FLUX deployment with min_replicas=0 to save cost. First request after 10 minutes idle takes 30 s instead of 2 s.

min_replicas=0 enables scale-to-zero, trading cold-start latency for idle cost. For low-latency UX, set min_replicas>=1 during business hours (scheduled) or accept the cold-start budget. Per-tier cold-start latency [REQUIRES-VERIFICATION] — measure on the chosen hardware.

Deployment with max_replicas=4 receives a burst of 200 concurrent requests; the 5th-200th queue.

Autoscaling scales up to max_replicas. Excess concurrent requests queue on the deployment. Tune max_replicas to absorb peak burst within an acceptable p95 latency. Above max_replicas, latency degrades — monitor queue depth as the leading indicator.

Operator wants p95 latency and request count for a deployment to drive alerting.

Use Replicate's deployment metrics surface (or scrape per-prediction metrics) and roll up in the operator's observability stack. Tag by deployment name and version. Alert on p95 latency drift relative to a rolling baseline, not absolute thresholds (which drift with model upgrades).