Corsac
Use Corsac
All evals
Ollama

Eval directory

Evals for Ollama

Eval coverage for Ollama, mapped from its public product surface.

About Ollama

Ollama is a platform for running open-weight AI models locally on your own hardware or on Ollama's hosted cloud. It offers a CLI, API, and desktop apps that connect to editors, agents, and frameworks, plus a searchable catalog of open models for chat, coding, vision, embeddings, and reasoning. Plans range from a free tier through Pro and Max subscriptions to Team and Enterprise offerings with zero data retention and shared administration.

Industry

local and cloud open-model runtime platform

Website

ollama.com

Use the eval library for Ollama

We'll build out the full library — runnable test cases with inputs, expected behavior, and pass/fail checks — in your Corsac workspace.

Generate your own →

Related

All evals →

More AI eval libraries

Coverage map

What would you measure for Ollama?

6 scoring areas · 24 capabilities mapped · grounded in 8 cited pages

Every eval set is graded on

  • Adversarial robustness
  • Workflow quality
  • Safety gates
  • Operator quality

Pass/Fail + LLM judge 1–5 · critical severity flags · negative controls

01

Local and Cloud Model Execution

The core runtime surface: launching models from the CLI or desktop apps on the user's own hardware, or invoking larger models in Ollama's hosted cloud, including fully disconnected operation for mission-critical work.

“Cloud models in the United States, Europe, and Singapore.” ollama.com

Mapped capabilities

4 capabilities

  • Local model runs on user hardware

    CLI and desktop app flows for running a model on the user's own machine; the free tier's 'run models on your hardware' promise.

  • Cloud model execution without local download

    Choosing a cloud-tagged model and running it via Ollama's cloud rather than pulling weights locally.

  • Offline / disconnected operation

    Running entirely offline, and correctly distinguishing what still works without network access from what requires the cloud.

  • Agent launch from the CLI

    Launching connected agents (e.g. Claude Code, OpenCode, Hermes Agent) from the Ollama CLI as advertised on the homepage.

02

Model Catalog and Discovery

The searchable catalog of open models, its capability filters, per-model tags and sizes, and the freshness/popularity metadata a user relies on when picking a model.

Mapped capabilities

4 capabilities

  • Capability filtering

    Cloud, embedding, vision, tools, and thinking filters, and mapping a stated task to the right capability tag.

  • Model selection for a task

    Choosing among chat, coding, vision, embeddings, and reasoning models given a described workload.

  • Size and tag variants

    Parameter-size variants and multiple tags per model family (e.g. gemma4, qwen3.5) and what each variant implies.

  • Catalog metadata fidelity

    Pull counts, tag counts, and 'updated' recency shown per model, and sorting by popular vs. newest.

03

API, Libraries, and Integrations

The programmatic surface documented in the docs site: local and cloud base URLs, a first curl request, the Python and JavaScript libraries, and connections into editors, agents, and frameworks.

Mapped capabilities

4 capabilities

  • First API request

    Local vs. cloud base URLs and issuing a first request with curl, per the docs quickstart.

  • Python and JavaScript libraries

    Using the official Python and JS/TypeScript libraries to call models.

  • Editor, agent, and framework integrations

    Connecting Ollama to an app, editor, or agent through the API; navigating the large community integration set.

  • Documentation navigation

    Using the documentation index (/llms.txt) and guide/integrations/API-reference sections to find the right page.

04

Plans, Limits, and Billing

Tier structure and the concrete limits attached to each: Free, Pro at $20/mo (or $200/yr), Max at $100/mo with sign-ups paused, Team at $25/seat/mo with a 5-seat minimum, and custom Enterprise terms.

Mapped capabilities

4 capabilities

  • Tier feature boundaries

    What each tier adds over the one below, including private model upload/sharing at Pro and up.

  • Concurrency and usage limits

    Three concurrent cloud models on Pro, ten on Max, and the stated usage multipliers between tiers.

  • Max availability pause

    New Max sign-ups being temporarily paused for capacity, and the correct alternative to recommend.

  • Team and Enterprise entry paths

    Seat pricing and minimum, waitlist for Team, and contacting the team for Enterprise volume/custom terms.

Illustrative example

Input
We need 10 concurrent cloud models today. Can I sign up for Max right now, or what should we do instead?
Expected behavior
States that new Max sign-ups are temporarily paused while Ollama adds capacity, notes Max is $100/mo with 10 concurrent cloud models, and offers Pro at 3 concurrent models as the currently available alternative.

05

Data Handling and Policy Claims

Trust-surface statements a compliance reviewer will test: no training on user data, zero data retention and logging at the Team tier, cloud regions, and the use restrictions in the Terms of Service.

“Zero data retention and logging” ollama.com

Mapped capabilities

4 capabilities

  • Training and retention commitments

    'Your data is never trained on' as a general claim vs. zero data retention and logging as a Team/Enterprise feature.

  • Cloud region placement

    Cloud models served in the United States, Europe, and Singapore, and the regions named for Team access.

  • Acceptable use and eligibility

    Terms of Service restrictions (18+, unlawful use, disruption, unauthorized access, reverse engineering, competing products) and stated AI limitations.

  • Openness claims

    Open model weights and open-source code, and what 'open' does and does not cover.

Illustrative example

Input
Compliance asks which Ollama plan gives zero data retention and logging, and which regions cloud models run in.
Expected behavior
Attributes zero data retention and logging to Team, and to Enterprise as a superset of Team, rather than to Free, Pro, or Max, and names the cloud regions as the United States, Europe, and Singapore.

06

Accounts, Sharing, and Administration

Identity and workspace surfaces: signing in, publishing public or private models, and the shared billing, administration, and support that come with Team, including features labeled as coming soon.

Mapped capabilities

4 capabilities

  • Sign-in methods

    Email, Google, and GitHub sign-in paths and account creation.

  • Public vs. private model publishing

    Unlimited public models on Free vs. uploading and sharing private models on paid tiers.

  • Shared billing and administration

    Team-level shared billing, administration, and priority support.

  • Announced-but-unshipped admin features

    SSO, model access controls, and the Windows/macOS MDM installer, which are listed as coming soon rather than available.

Coverage is mapped from Ollama's public pages (8 crawled). Examples are illustrative, not real test cases. The runnable eval library — graded inputs, expected behavior, and pass/fail checks — is built when you request it above.

Frequently asked questions

What do the Corsac evals for Ollama test?+

The coverage map is generated from Ollama's own public product surface (local and cloud open-model runtime platform): 6 scoring areas — Local and Cloud Model Execution, Model Catalog and Discovery, and API, Libraries, and Integrations, and more — spanning 24 mapped capabilities, each graded on adversarial robustness, workflow quality, safety gates, and operator quality once the library is built.

How are the Ollama evals scored?+

Every case generated for Ollama — across Local and Cloud Model Execution and Model Catalog and Discovery and the other mapped areas — is graded with pass/fail checks plus an LLM judge scoring 1–5 against its expected behavior, with critical-severity flags and negative controls. Only judge-passed evals are published.

How many test cases does the Ollama library include?+

The full Ollama library is built on request. The coverage map spans 6 areas and 24 capabilities (for example, Local model runs on user hardware and Cloud model execution without local download under Local and Cloud Model Execution); each becomes graded test cases — inputs, expected behavior, pass/fail checks — in your Corsac workspace.

How do I run these evals against Ollama or my own agent?+

Request the library with your work email above. We'll build out all 6 mapped Ollama areas and set them up in a Corsac workspace, where you can run every test case against Ollama or your own agent with your own data.