Cohere
For CohereAI Platform

Fine Tuning And Customization

Cohere API · Cohere

Foundation Model & API — Cohere

Evaluates Cohere's Fine-tuning & Customization across 9 scenario-based test cases, each graded against an expected-behavior rubric by an LLM judge, from Corsac's Foundation Model & API eval coverage.

About Cohere

Cohere builds enterprise foundation models and the tools around them — the Command model family, best-in-class Rerank and Embed endpoints, and grounded retrieval-augmented generation with inline citations — deployable across major clouds and private VPCs.

Employees

~400

Industry

Foundation Model

Headquarters

Toronto, Canada

Website

cohere.com

Sample tests· showing 3 of 9

#InputExpected behaviorCheck
01

Operator uploads a fine-tuning dataset for a chat model in a free-form CSV instead of the documented JSONL turn format and expects the job to start.

Format the training data to the documented schema for the fine-tune type (e.g., JSONL chat turns for chat fine-tuning) and validate it before submission. A schema-invalid dataset should fail fast at upload, not after a long training run.

Pass / FailAi Platformhigh
02

An operator fine-tunes a classifier and evaluates it on the same examples used for training, reporting near-perfect accuracy.

Hold out a validation set the model never trains on and report metrics on the held-out set; provide a separate validation file where the fine-tuning API supports it. Same-set evaluation overstates quality and hides overfitting.

Pass / FailAi Platformhigh
03

Operator creates a chat fine-tune but intends to use it for the rerank endpoint, expecting one fine-tune to serve every endpoint.

Choose the fine-tune type that matches the target endpoint (chat, classify, or rerank); a fine-tuned model is used on the endpoint it was trained for. Do not assume a chat fine-tune is callable as a reranker.

Pass / FailAi Platformhigh

Unlock full benchmark

6 more test cases

Use this benchmark

How this eval is graded

Grade against expected.ideal_behavior and expected.rubric. Per-criterion pass requires mean >= 4.0 and no criterion below 3.

Rubric criteria

  • Cohere
  • Ai Platform
  • Fine Tuning And Customization

Recommended for

Cohere APICohere customers

Works with

Related evals

Frequently asked questions

What does the Fine Tuning And Customization eval for Cohere Cohere API test?+

Evaluates Cohere's Fine-tuning & Customization across 9 scenario-based test cases, each graded against an expected-behavior rubric by an LLM judge, from Corsac's Foundation Model & API eval coverage.

How is the Fine Tuning And Customization eval scored?+

The judge rubric: Grade against expected.ideal_behavior and expected.rubric. Per-criterion pass requires mean >= 4.0 and no criterion below 3.

How many test cases does this eval pack include?+

The Fine Tuning And Customization pack for Cohere Cohere API contains 9 test cases. 3 sample cases are shown free on this page; the full set runs in a Corsac workspace.

How do I run this eval?+

Sign up for Corsac, connect your model or agent endpoint, and run the Fine Tuning And Customization pack as-is or after customizing thresholds. Results land in your workspace with per-case scores, and you can gate releases on the pack in CI via the REST API.

Run this eval in your workspace

Connect your data, configure thresholds, and review results with your team.