Eval directory

Evals for Oumi

Eval coverage for Oumi, mapped from its public product surface.

About Oumi

Oumi is an AI-native platform that automates the custom model development lifecycle — dataset building, evaluation, training (SFT, PEFT/LoRA, on-policy distillation), and deployment — so teams can go from prompt to a deployed specialized model in hours. It positions owned, task-specific models as more accurate and far cheaper than renting frontier models, with customers keeping their own weights and data. Offered in Free, Pro ($25/month pay-as-you-go), and Enterprise tiers, the latter adding BYOC/VPC and on-prem deployment, dedicated GPU capacity, and SLAs.

Industry

custom AI model development platform

Headquarters

Bellevue, Washington

Website

oumi.ai

Use the eval library for Oumi

We'll build out the full library — runnable test cases with inputs, expected behavior, and pass/fail checks — in your Corsac workspace.

All evals →

More AI eval libraries

Coverage map

What would you measure for Oumi?

6 scoring areas · 24 capabilities mapped · grounded in 8 cited pages

Every eval set is graded on

Adversarial robustness
Workflow quality
Safety gates
Operator quality

Pass/Fail + LLM judge 1–5 · critical severity flags · negative controls

Agentic Lifecycle Automation

The Oumi agent's automation of the end-to-end loop — evaluate, analyze failures, curate data, train, redeploy — and how clearly the product explains what it does autonomously versus what the user decides.

“as low as 2 hours to production from prompt to deploy” oumi.ai

Mapped capabilities

4 capabilities

End-to-end run from a prompt
Taking a task description through dataset, eval, training, and deploy stages as a single automated flow.
Iteration loop orchestration
Sequencing eval -> failure analysis -> data curation -> retraining, and deciding when another iteration is warranted.
Automation scope and user control
Communicating which steps the agent performs on its own and where the user reviews, overrides, or approves.
Time-to-production expectations
Setting realistic run-duration expectations against the stated hours-not-months positioning.

Dataset Building and Synthesis

Constructing and refining the training dataset, including synthetic data generation with open or closed models plus analysis and curation of what is kept.

“Customer prompts, inputs, outputs, inference results, evaluation data, and Customer models are your Confidential Information.” oumi.ai

Mapped capabilities

4 capabilities

Data synthesis with open/closed models
Generating rows for a described task and choosing the generating model.
Data analysis and curation
Inspecting a candidate dataset and filtering or reshaping it before training.
Dataset-to-training handoff
Producing a dataset in a form the SFT/PEFT stages consume, including token-count accounting.
Synthesis and storage metering
Explaining row-based synthesis and GB-month storage usage against plan credits.

Evaluation and Failure Analysis

Measuring model quality on the user's task, surfacing failure modes, and feeding those findings back into data and training decisions.

Mapped capabilities

4 capabilities

Open and closed model evaluation
Running comparable evaluations across candidate base models and a custom model.
Failure mode identification
Grouping incorrect outputs into named, actionable failure categories rather than a single score.
Judgment-based metering
Relating evaluation volume to judgment counts and their per-1,000 pricing.
Baseline comparison framing
Presenting custom-model results against a frontier baseline without overstating the delta.

Training and Customization

The supported training methods and the guidance for choosing among them given task, model size, budget, and plan tier.

Mapped capabilities

4 capabilities

Supervised fine-tuning (SFT)
Running SFT and reasoning about tokens, epochs, and model-size bands.
PEFT: LoRA and QLoRA
Parameter-efficient tuning options and when they are preferable to full SFT.
On-policy distillation
Distillation runs on dedicated GPUs, their tier requirement, and GPU-hour cost basis.
Method selection guidance
Recommending a training method for a stated task and constraint set, including Enterprise-only advanced methods.

Illustrative example

Input: I'm on the Free plan. Can I run on-policy distillation there, and how is it billed?
Expected behavior: States that on-policy distillation is not in Free and requires Pro or above, notes Free covers SFT and PEFT (LoRA, QLoRA), and explains distillation is billed per GPU-hour on dedicated GPUs rather than per training token.

Deployment, Inference and Model Portability

Getting a trained model into production and keeping it portable — autoscaled inference, concurrent jobs, weight downloads, and running outside Oumi's hosted platform.

“Download model weights: 1 / month” oumi.ai

Mapped capabilities

4 capabilities

Production inference with autoscaling
Deploying a trained model to a scaling inference endpoint.
Concurrent job execution
Running multiple training or evaluation jobs in parallel where the plan allows it.
Weight download and run-anywhere
Exporting owned weights, including the Pro monthly download allowance, and running them elsewhere.
Dedicated capacity and throughput
Enterprise dedicated GPU capacity, scaling to large fleets, and guaranteed-throughput commitments.

Plans, Pricing and Data Governance

The commercial and trust envelope: what each tier includes, how usage-based charges are computed, how customer content is treated versus telemetry, and enterprise deployment controls.

Mapped capabilities

4 capabilities

Tier entitlements
Which capabilities belong to Free, Pro, and Enterprise, and what each tier's credits cover.
Usage-based pricing arithmetic
Computing SFT, evaluation, synthesis, and storage charges from published rates.
Customer content vs. telemetry
Distinguishing confidential customer prompts, datasets, weights, and eval artifacts from operational telemetry and metadata.
Enterprise deployment controls
BYOC/VPC and on-prem options, SLAs, and embedded support scope.

Illustrative example

Input: I want to fine-tune a 24B model on a 3M-token dataset for 2 epochs on your hosted platform. What will the supervised fine-tuning cost?
Expected behavior: Places 24B in the 16.1-32B band at $2.00 per 1M training tokens, multiplies the 3M-token dataset by 2 epochs to get 6M billed tokens, and gives $12.00 for the training run.

Coverage is mapped from Oumi's public pages (8 crawled). Examples are illustrative, not real test cases. The runnable eval library — graded inputs, expected behavior, and pass/fail checks — is built when you request it above.

Frequently asked questions

What do the Corsac evals for Oumi test?+

The coverage map is generated from Oumi's own public product surface (custom AI model development platform): 6 scoring areas — Agentic Lifecycle Automation, Dataset Building and Synthesis, and Evaluation and Failure Analysis, and more — spanning 24 mapped capabilities, each graded on adversarial robustness, workflow quality, safety gates, and operator quality once the library is built.

How are the Oumi evals scored?+

Every case generated for Oumi — across Agentic Lifecycle Automation and Dataset Building and Synthesis and the other mapped areas — is graded with pass/fail checks plus an LLM judge scoring 1–5 against its expected behavior, with critical-severity flags and negative controls. Only judge-passed evals are published.

How many test cases does the Oumi library include?+

The full Oumi library is built on request. The coverage map spans 6 areas and 24 capabilities (for example, End-to-end run from a prompt and Iteration loop orchestration under Agentic Lifecycle Automation); each becomes graded test cases — inputs, expected behavior, pass/fail checks — in your Corsac workspace.

How do I run these evals against Oumi or my own agent?+

Request the library with your work email above. We'll build out all 6 mapped Oumi areas and set them up in a Corsac workspace, where you can run every test case against Oumi or your own agent with your own data.