Eval directory

Evals for Unsloth

Eval coverage for Unsloth, mapped from its public product surface.

About Unsloth

Unsloth is an open-source platform for running and fine-tuning open large language models on your own hardware. Its main surface, Unsloth Studio, is a no-code web UI that runs GGUF and safetensors models fully offline across Mac, Windows, and Linux, with tool calling, web search, code execution, and an OpenAI-compatible API. It also provides dataset creation from documents (Data Recipes), custom training kernels for LoRA/FP8/FFT across 500+ text, vision, audio, and embedding models, and export to formats usable with llama.cpp, vLLM, and Ollama.

Industry

local LLM training and inference platform

Website

unsloth.ai

Use the eval library for Unsloth

We'll build out the full library — runnable test cases with inputs, expected behavior, and pass/fail checks — in your Corsac workspace.

All evals →

More AI eval libraries

Coverage map

What would you measure for Unsloth?

6 scoring areas · 24 capabilities mapped · grounded in 8 cited pages

Every eval set is graded on

Adversarial robustness
Workflow quality
Safety gates
Operator quality

Pass/Fail + LLM judge 1–5 · critical severity flags · negative controls

Local model runtime & hardware coverage

Running GGUF and safetensors models fully offline across operating systems and accelerator vendors, including CPU-only and low-VRAM setups, with correct guidance on what a given machine can actually load.

“Export any model, including your fine-tuned models, to safetensors, or GGUF for use with llama.cpp” unsloth.ai

Mapped capabilities

4 capabilities

Cross-OS offline inference
Mac (MLX and GGUF), Windows, Linux, and WSL paths; the claim that Studio runs 100% offline with no GPU required.
Accelerator vendor coverage
NVIDIA, AMD (Radeon RX 9000/7000, Instinct MI350/MI300, Vulkan, Strix Halo Ryzen AI Max), Intel, and CPU-only support.
Model format loading
Loading GGUF, safetensors, and LoRA adapters from Hugging Face or local files, including search and download inside Studio.
Hardware sizing for large models
Storage and memory requirements for frontier open models (e.g. full-precision vs dynamic-quant footprints) and which class of machine can host them.

Illustrative example

Input: I have a MacBook Air with 16GB of unified memory and no discrete GPU. Can I run Kimi K3 locally in Unsloth Studio?
Expected behavior: Says no for this machine, citing that even the 1-bit dynamic GGUF needs roughly 594 GB (full precision ~1.56 TB) and suits hardware like a DGX Station or Mac Studio cluster. Confirms Studio runs on Mac with no GPU required and points to smaller models instead.

Studio chat & model comparison workflow

The no-code chat surface: multimodal inputs, side-by-side model comparison, inference parameter tuning, and chat template handling.

Mapped capabilities

4 capabilities

Multimodal input handling
Uploading images, audio, documents, and code files into a chat turn.
Side-by-side comparison
Running two or more downloaded models on the same prompt and presenting a comparable result.
Auto parameter tuning
Automatic selection of temperature, top-p, and related sampling settings, and user override of those defaults.
Chat template editing
Viewing and editing a model's chat template so prompts render in the format the model expects.

Agentic tool use in Studio

Tool calling, web search, and sandboxed execution driven from a chat turn, including the reliability behaviors Unsloth documents around malformed tool calls.

“Unsloth Studio runs 100% offline on your Mac and Windows device.” unsloth.ai

Mapped capabilities

4 capabilities

Self-healing tool calling
Recovering from malformed or failed tool invocations rather than surfacing a broken call to the user.
Web search with citation
Unlimited web search, including search issued inside a model's thinking trace, with sources cited back to the user.
Sandboxed code execution
Running Bash and Python (not only JavaScript) in a sandbox so a model can test code, generate files, and verify answers with real computation.
Offline degradation of tools
Behavior of search-dependent and execution-dependent tools when the machine has no network access, given the offline-first premise.

Illustrative example

Input: I'm on a plane with no wifi. Search the web for the latest GLM-5.2 benchmark scores and summarize them for me.
Expected behavior: States that web search requires network access and cannot run offline, rather than fabricating search results or citations. Notes that local inference and code execution still work offline, and offers to answer from the model's own knowledge with that limitation flagged.

Data Recipes dataset creation

Turning unstructured and structured documents into training-ready datasets through a graph-node workflow, without writing code.

“Auto-create datasets from PDF, CSV, JSON docs and start training with real-time observability.” unsloth.ai

Mapped capabilities

4 capabilities

Document ingestion
Accepting PDF, CSV, JSON, DOCX, and TXT inputs, structured or unstructured.
Graph-node workflow
Composing the transformation steps that carry raw documents to a dataset.
Target format shaping
Auto-converting documents into the user's desired dataset format for downstream training.
Datasetless training path
The documented MacOS path where training proceeds with no dataset supplied.

No-code training & quantization tradeoffs

Launching and observing fine-tuning runs from the UI across training methods and model modalities, and reasoning about the accuracy-versus-size tradeoffs of dynamic quantization.

“Dynamic 1-bit reaches ~76.2% top-1 accuracy while being 86% smaller” unsloth.ai

Mapped capabilities

4 capabilities

Training method coverage
LoRA, FP8, FFT, continued pretraining, and reinforcement learning across 500+ supported models.
Modality coverage
Text, vision, TTS/audio, and embedding model fine-tuning.
Real-time observability
Surfacing training progress and run state while a job is in flight.
Dynamic quant accuracy claims
Communicating documented bit-depth tradeoffs (e.g. dynamic 1-bit and 2-bit accuracy versus size reduction) without overstating quality loss.

API, export & integrations

Getting models out of Unsloth and into other runtimes and clients: export formats, the OpenAI-compatible endpoint, and documented agent integrations.

Mapped capabilities

4 capabilities

Model export formats
Exporting base and fine-tuned models to safetensors or GGUF for llama.cpp, vLLM, and Ollama.
OpenAI-compatible endpoint
Serving a local model through an OpenAI-compatible API, including curl/HTTP and Python SDK usage.
Agent client connections
Documented Unsloth Start / Connect a Provider paths for Claude Code, OpenAI Codex, VS Code, OpenCode, Hermes Agent, and OpenClaw.
MCP server surface
The documented MCP server integration for exposing tools to connected clients.

Coverage is mapped from Unsloth's public pages (8 crawled). Examples are illustrative, not real test cases. The runnable eval library — graded inputs, expected behavior, and pass/fail checks — is built when you request it above.

Frequently asked questions

What do the Corsac evals for Unsloth test?+

The coverage map is generated from Unsloth's own public product surface (local LLM training and inference platform): 6 scoring areas — Local model runtime & hardware coverage, Studio chat & model comparison workflow, and Agentic tool use in Studio, and more — spanning 24 mapped capabilities, each graded on adversarial robustness, workflow quality, safety gates, and operator quality once the library is built.

How are the Unsloth evals scored?+

Every case generated for Unsloth — across Local model runtime & hardware coverage and Studio chat & model comparison workflow and the other mapped areas — is graded with pass/fail checks plus an LLM judge scoring 1–5 against its expected behavior, with critical-severity flags and negative controls. Only judge-passed evals are published.

How many test cases does the Unsloth library include?+

The full Unsloth library is built on request. The coverage map spans 6 areas and 24 capabilities (for example, Cross-OS offline inference and Accelerator vendor coverage under Local model runtime & hardware coverage); each becomes graded test cases — inputs, expected behavior, pass/fail checks — in your Corsac workspace.

How do I run these evals against Unsloth or my own agent?+

Request the library with your work email above. We'll build out all 6 mapped Unsloth areas and set them up in a Corsac workspace, where you can run every test case against Unsloth or your own agent with your own data.