Eval directory

Evals for Latitude.sh

Eval coverage for Latitude.sh, mapped from its public product surface.

About Latitude.sh

Latitude.sh is a global cloud platform for deploying and managing automated bare metal servers, dedicated GPU clusters, virtual machines, and Kubernetes clusters across 25 locations. It pairs dedicated hardware performance with a self-serve platform accessible via dashboard, CLI (lsh), and a JSON:API-conformant API. Supporting products include object/block/file storage, managed PostgreSQL, a server-level firewall, and private connectivity to other clouds.

Industry

global bare metal and GPU cloud infrastructure

Website

www.latitude.sh

Use the eval library for Latitude.sh

We'll build out the full library — runnable test cases with inputs, expected behavior, and pass/fail checks — in your Corsac workspace.

All evals →

More AI eval libraries

Coverage map

What would you measure for Latitude.sh?

6 scoring areas · 23 capabilities mapped · grounded in 8 cited pages

Every eval set is graded on

Adversarial robustness
Workflow quality
Safety gates
Operator quality

Pass/Fail + LLM judge 1–5 · critical severity flags · negative controls

Bare Metal & GPU Compute

Deploying and managing automated bare metal servers and dedicated GPU clusters, including plan/spec selection, location choice, and the hourly vs. monthly vs. yearly billing options surfaced in pricing.

“Deploy and manage fully automated bare metal servers globally in under 5 seconds.” www.latitude.sh

Mapped capabilities

4 capabilities

Plan and spec selection
Matching CPU/RAM/NVMe/network specs (e.g. rs4.metal.xlarge, c4.metal.large) to a stated workload.
GPU cluster configuration
Reasoning over GPU plans, VRAM per GPU vs. total, and interconnect details (e.g. g4.b300.large, g3.h100.small).
Location and global footprint
Choosing among the 25 locations and explaining per-plan location availability.
Billing mode trade-offs
Hourly vs. monthly vs. paid-yearly pricing and included bandwidth (20 TB free out / unlimited in).

Virtual Machines & Kubernetes

General-purpose VM deployment and RKE2-based Kubernetes clusters on bare metal, covering creation flows, access, scaling limits, and the documented split of responsibilities.

“Latitude.sh Kubernetes provisions RKE2 clusters on dedicated bare metal.” www.latitude.sh

Mapped capabilities

4 capabilities

VM deployment flow
Plan/location/OS selection, multi-instance deploys (1-5) and the auto-generated naming pattern.
Instance access and power state
SSH access with provided username/password, disabled root login, and power-cycle behavior including billing implications.
Cluster provisioning and access
RKE2 cluster creation prerequisites, kubeconfig download, and BGP/MetalLB LoadBalancer IPs.
Cluster scaling and shared responsibility
Control plane (1-3) and worker (0-10) resizing, version upgrades, and Latitude.sh vs. customer day-2 duties.

Illustrative example

Input: On a Latitude.sh Kubernetes cluster, who is responsible for ingress, monitoring, backups, and Kubernetes version upgrades?
Expected behavior: States that Latitude.sh delivers the running RKE2 cluster — control plane, worker nodes, kubeconfig, and a BGP-announced LoadBalancer IP — while the customer owns day-2 operations including ingress, storage classes, monitoring, backups, node OS patching, and Kubernetes version upgrades.

API & CLI Automation

Programmatic control of the platform through the JSON:API-conformant REST API and the open-source lsh command line interface.

Mapped capabilities

4 capabilities

CLI authentication and profiles
Browser-assisted login vs. --with-token, per-team profiles, profile use, and auth status.
CLI installation and output modes
Homebrew and install-script setup; table output vs. --json for scripting.
JSON:API conventions
Resource shapes and conformance to the JSON:API specification for generalized tooling.
Filtering and sorting
Using built-in filtering and sorting to query infrastructure programmatically.

Illustrative example

Input: I'm scripting Latitude.sh deploys in CI, so there's no browser. How do I authenticate lsh, and where do I get the credential?
Expected behavior: Recommends API key authentication for headless use: create the key from Settings → API Keys in the dashboard, then run lsh login --with-token <API_KEY>. May note browser-assisted lsh login as the interactive alternative, without inventing other flags.

Networking, Firewall & Connectivity

Server-level firewall management, private interconnection to other clouds, and network observability as documented for the platform.

Mapped capabilities

4 capabilities

Firewall rules management
Creating firewalls and configuring inbound/outbound rules (From/To, protocol, port) from a single interface.
Firewall prerequisites and scope
UFW-compatible OS requirement, agent installation, and server-level vs. perimeter enforcement.
Server assignment and lifecycle
Assigning or removing protected servers and the Overview/Rules/Servers/Settings tab model.
Private cloud connectivity
Cloud Gateway private connections from Latitude.sh servers to AWS, GCP, Azure, and others.

Storage & Managed Data

NVMe-backed storage products and fully managed PostgreSQL, including how each is positioned and priced relative to compute.

Mapped capabilities

3 capabilities

Storage type selection
Choosing among Object, Block, and File storage for a described workload.
Managed PostgreSQL
Positioning and scope of fully managed PostgreSQL instances alongside Latitude.sh infrastructure.
Storage and data pricing
Locating storage and Postgres pricing under the Storage & Data pricing category.

Account Setup & Support Workflows

The prerequisites, self-serve onboarding steps, and support/status surfaces a customer moves through before and after deployment.

Mapped capabilities

4 capabilities

Deployment prerequisites
Verified account, payment method on file, at least one team/project SSH key, and a selected project.
Project and team organization
Selecting a project and working across teams, including per-team CLI profiles and API keys.
Status, changelog and escalation
Routing to the status page, product changelog, support contact, or sales contact as appropriate.
Custom and enterprise paths
Directing demanding or unlisted needs to Build, the GPU wishlist, or the sales team.

Coverage is mapped from Latitude.sh's public pages (8 crawled). Examples are illustrative, not real test cases. The runnable eval library — graded inputs, expected behavior, and pass/fail checks — is built when you request it above.

Frequently asked questions

What do the Corsac evals for Latitude.sh test?+

The coverage map is generated from Latitude.sh's own public product surface (global bare metal and GPU cloud infrastructure): 6 scoring areas — Bare Metal & GPU Compute, Virtual Machines & Kubernetes, and API & CLI Automation, and more — spanning 23 mapped capabilities, each graded on adversarial robustness, workflow quality, safety gates, and operator quality once the library is built.

How are the Latitude.sh evals scored?+

Every case generated for Latitude.sh — across Bare Metal & GPU Compute and Virtual Machines & Kubernetes and the other mapped areas — is graded with pass/fail checks plus an LLM judge scoring 1–5 against its expected behavior, with critical-severity flags and negative controls. Only judge-passed evals are published.

How many test cases does the Latitude.sh library include?+

The full Latitude.sh library is built on request. The coverage map spans 6 areas and 23 capabilities (for example, Plan and spec selection and GPU cluster configuration under Bare Metal & GPU Compute); each becomes graded test cases — inputs, expected behavior, pass/fail checks — in your Corsac workspace.

How do I run these evals against Latitude.sh or my own agent?+

Request the library with your work email above. We'll build out all 6 mapped Latitude.sh areas and set them up in a Corsac workspace, where you can run every test case against Latitude.sh or your own agent with your own data.