Eval directory

Evals for GMI Cloud

Eval coverage for GMI Cloud, mapped from its public product surface.

About GMI Cloud

GMI Cloud is an AI-native inference cloud built on NVIDIA GPU platforms, offering serverless inference alongside dedicated bare-metal GPU clusters. It provides an OpenAI-compatible API endpoint with access to 200+ models, plus Python/TypeScript SDKs and an AgentBox managed compute product for agents. GPUs including H100, H200, B200, GB200 NVL72 and GB300 NVL72 are offered on-demand or through reserved capacity with published per-GPU-hour pricing.

Industry

GPU cloud & AI inference infrastructure

Website

www.gmicloud.ai

Use the eval library for GMI Cloud

We'll build out the full library — runnable test cases with inputs, expected behavior, and pass/fail checks — in your Corsac workspace.

All evals →

More AI eval libraries

Coverage map

What would you measure for GMI Cloud?

6 scoring areas · 24 capabilities mapped · grounded in 8 cited pages

Every eval set is graded on

Adversarial robustness
Workflow quality
Safety gates
Operator quality

Pass/Fail + LLM judge 1–5 · critical severity flags · negative controls

OpenAI-Compatible Inference API & SDKs

Correct use of the single base URL, key handling, request shape, and streaming across the documented routes, including the Python/TypeScript SDKs as drop-in replacements for the OpenAI client and the AgentBox managed-compute surface for agents.

Mapped capabilities

4 capabilities

Base URL, auth header, and endpoint routing
https://api.gmi-serving.com, Bearer key from console, POST /v1/chat/completions and sibling routes
Drop-in OpenAI SDK migration
Swapping base_url/api_key while keeping request shape and model-id semantics unchanged
Streaming responses
stream:true request flag and reading incremental chunk deltas
AgentBox agent deployment
Managed compute for running agents; when to reach for it versus the raw inference API

Illustrative example

Input: I have a working Python script using the OpenAI client with streaming. What is the smallest change to run it against GMI Cloud instead?
Expected behavior: Says to set base_url to https://api.gmi-serving.com/v1 and pass a GMI API key from the console, then use a model id from the GMI catalog. Confirms the request shape and streaming loop stay unchanged; no new SDK or custom parameters are required.

Model Catalog & Selection

Helping a user find, name, and switch models across the 200+ catalog without changing surrounding code — including how model ids are discovered and how chat, vision, and reasoning workloads map onto available models.

Mapped capabilities

4 capabilities

Model discovery via GET /v1/models
Listing available ids rather than guessing model names
Model-id substitution as the switching mechanism
'Keep your code, change the model ID' — no other call-site changes implied
Chat, vision, and reasoning workload fit
Routing a described workload to the documented modality categories
Open-weight and newly launched model availability
Day-0 and open-weight launches as covered on the blog, stated with dates and without inventing benchmark numbers

GPU Platform Selection & Provisioning

Choosing among the published NVIDIA platforms and deployment modes for a stated workload, and explaining what dedicated infrastructure gives you that serverless does not.

“combining serverless scaling and dedicated GPU infrastructure with predictable performance and cost” www.gmicloud.ai

Mapped capabilities

4 capabilities

Platform fit by workload
H100, H200, B200, GB200 NVL72, GB300 NVL72 mapped to the documented ideal workloads (long-context, large-batch inference, distributed training)
Bare metal and root access
Full root control, custom stacks, container platforms
Multi-node cluster orchestration
Cluster engine at the infrastructure layer; high-bandwidth interconnect for NVL72 systems
On-demand versus reserved capacity
Elastic on-demand access versus reserved plans for predictable long-term workloads

Pricing, Availability & Commitments

Reading the published price sheet accurately: per-GPU-hour floors, arithmetic across multi-GPU configurations, and the distinct availability states, without inventing discounts or quoting a price for hardware that isn't generally available.

“Automatic scaling to zero with no idle cost” www.gmicloud.ai

Mapped capabilities

4 capabilities

Published per-GPU-hour rates
H100 $2.00, H200 $2.60, B200 $4.00, GB200 NVL72 $8.00; 'from' framing preserved
Availability state fidelity
Available Now vs Limited Availability (B200) vs Pre-order (GB300 NVL72, no listed rate)
Commitment and reserved-capacity savings
Commitment-based structures and usage-adaptive on-demand → committed transitions, described qualitatively as published
Cost arithmetic and scale-to-zero economics
Multi-GPU hourly totals; no idle cost under serverless autoscaling

Illustrative example

Input: What would eight H100s cost me per hour on-demand, and can I spin up a GB300 NVL72 cluster this week instead?
Expected behavior: Quotes H100 at $2.00 per GPU-hour, giving $16.00 per hour for eight, and notes this is a 'from' starting rate. States that GB300 NVL72 is pre-order only with no published per-GPU-hour price, so it cannot be provisioned this week.

Serverless Scaling & Performance Behavior

Explaining the documented runtime behaviors that determine latency and cost under load, and guiding the transition from shared serverless to a dedicated endpoint when performance needs to be fixed.

“Dedicated NVIDIA GPU resources, no shared environments or performance variability.” www.gmicloud.ai

Mapped capabilities

4 capabilities

Automatic scaling to zero
No idle cost; implications for cold traffic patterns
Batching and latency-aware scheduling
Built-in throughput and latency controls at the platform layer
Multi-tenant isolation and dedicated endpoints
Predictable performance claims; switching to a dedicated endpoint for fixed latency
Published reliability and footprint targets
99.9% uptime target and global data center coverage, cited as targets rather than guarantees

Policy, Compliance & Service Terms

Directing users to the correct published agreement or policy for a given question and declining to improvise contractual, export-control, or acceptable-use answers that the legal pages do not state.

Mapped capabilities

4 capabilities

Agreement routing
Services Agreement, SLA, Reseller Agreement, Proof of Concept Agreement, MaaS Console Terms of Service
Sanctions and export control questions
Points to the published compliance policy; no improvised jurisdiction or eligibility rulings
Acceptable use and user content
AUP and User Content Disclaimer as the governing documents for workload restrictions
Privacy and data handling inquiries
Routes to the published privacy policy instead of asserting retention or training-data terms

Coverage is mapped from GMI Cloud's public pages (8 crawled). Examples are illustrative, not real test cases. The runnable eval library — graded inputs, expected behavior, and pass/fail checks — is built when you request it above.

Frequently asked questions

What do the Corsac evals for GMI Cloud test?+

The coverage map is generated from GMI Cloud's own public product surface (GPU cloud & AI inference infrastructure): 6 scoring areas — OpenAI-Compatible Inference API & SDKs, Model Catalog & Selection, and GPU Platform Selection & Provisioning, and more — spanning 24 mapped capabilities, each graded on adversarial robustness, workflow quality, safety gates, and operator quality once the library is built.

How are the GMI Cloud evals scored?+

Every case generated for GMI Cloud — across OpenAI-Compatible Inference API & SDKs and Model Catalog & Selection and the other mapped areas — is graded with pass/fail checks plus an LLM judge scoring 1–5 against its expected behavior, with critical-severity flags and negative controls. Only judge-passed evals are published.

How many test cases does the GMI Cloud library include?+

The full GMI Cloud library is built on request. The coverage map spans 6 areas and 24 capabilities (for example, Base URL, auth header, and endpoint routing and Drop-in OpenAI SDK migration under OpenAI-Compatible Inference API & SDKs); each becomes graded test cases — inputs, expected behavior, pass/fail checks — in your Corsac workspace.

How do I run these evals against GMI Cloud or my own agent?+

Request the library with your work email above. We'll build out all 6 mapped GMI Cloud areas and set them up in a Corsac workspace, where you can run every test case against GMI Cloud or your own agent with your own data.