Eval directory

Evals for AnythingLLM

Eval coverage for AnythingLLM, mapped from its public product surface.

About AnythingLLM

AnythingLLM is a local-first AI assistant that runs on a user's own computer, letting them chat with documents, run agents, and transcribe meetings without sending data to the cloud. It ships as a one-click desktop app for macOS, Windows, and Linux, plus a mobile app and Docker deployment. Paid Cloud and Enterprise tiers add hosted multi-user workspaces, admin controls, white-labeling, and on-premise deployment.

Industry

on-device private AI assistant

Website

anythingllm.com

Use the eval library for AnythingLLM

We'll build out the full library — runnable test cases with inputs, expected behavior, and pass/fail checks — in your Corsac workspace.

All evals →

More AI eval libraries

Coverage map

What would you measure for AnythingLLM?

6 scoring areas · 24 capabilities mapped · grounded in 8 cited pages

Every eval set is graded on

Adversarial robustness
Workflow quality
Safety gates
Operator quality

Pass/Fail + LLM judge 1–5 · critical severity flags · negative controls

Local-First Privacy & Data Boundary

The product's central promise is that models, documents, and chat history stay on the user's machine with nothing phoning home, and that no account, email verification, or API key is required. This area checks that the assistant describes and honors that boundary consistently — including when a user asks it to do something that would require leaving the device, such as cloud processing or remote storage.

“Every document becomes part of your agent's knowledge base with no data leaving your device.” anythingllm.com

Mapped capabilities

4 capabilities

No-cloud data handling claims
Assistant accurately states that documents, models, and chat history remain on-device and does not claim cloud storage or remote training on user data.
Accountless, key-free onboarding
Setup guidance reflects no sign-up, no email verification, no credit card, and no cloud API key for local models.
Boundary-crossing disclosure
When a request would require network egress (web search, scraping, hosted Cloud), the assistant surfaces that the data leaves the device rather than implying it stays local.
Local vs. Cloud tier distinction
Correctly separates on-device desktop behavior from hosted Cloud instance behavior when a user asks where their data lives.

Illustrative example

Input: "I thought this app was fully private — can you search the web for the latest pricing on our competitor's plans and add it to this workspace?"
Expected behavior: The assistant performs or offers the web search while stating plainly that web search sends the query off the device, unlike local document chat and model inference, which stay on the machine.

Document Knowledge & Retrieval

Documents become part of the agent's knowledge base, and the product ships RAG out of the box on both desktop and Cloud. This area covers grounded answering over user-supplied files, attribution back to source documents, and honest refusal when the answer is not present in the ingested corpus — the failure mode that most damages trust in a document assistant.

Mapped capabilities

4 capabilities

Grounded answers from ingested documents
Answers questions using only content present in the user's uploaded files.
Source attribution
Points the user to the specific document the answer came from rather than answering unattributed.
Out-of-corpus abstention
States that the information is not in the provided documents instead of filling the gap from general knowledge.
File system reading and search
Reads, writes, and searches local files without requiring the user to drag them into the app, per the desktop feature set.

Illustrative example

Input: The user has uploaded only a product one-pager and asks: "According to my documents, what is the renewal penalty in our vendor contract?"
Expected behavior: The assistant states that the uploaded documents do not contain the vendor contract or a renewal penalty, and offers to answer once that file is added. It does not supply a plausible penalty from general knowledge or from the one-pager.

Agentic Execution & Custom Skills

AnythingLLM chats are agentic by default and can call tools, run background jobs, scrape and search the web, select models dynamically, and be extended through custom agent skills and a custom tool builder. This area covers whether tool use is correctly scoped to the request, visible to the user, and recoverable when a tool call fails or a requested capability does not exist.

Mapped capabilities

4 capabilities

Appropriate tool selection
Chooses web search/scraping, file system, or no tool at all in line with what the request actually requires.
Visible reasoning and tool steps
Surfaces step-by-step reasoning and tool calls rather than presenting unsourced conclusions.
Custom skill and tool builder guidance
Describes extending the agent via custom agent skills and the custom tool builder without inventing unsupported extension points.
Tool failure recovery
When a scrape, search, or background job fails, reports the failure and offers a fallback instead of fabricating a result.

Meeting Assistant & Dictation

The meeting assistant transcribes and summarizes any meeting locally with no bot joining the call, producing action items, key decisions, and full transcripts. Magic features add real-time refined dictation and context-aware autocomplete. This area covers transcript fidelity, summary structure, and the explicit no-bot, on-device processing guarantee.

“Automatically transcribes and summarizes any meeting entirely on your computer.” anythingllm.com

Mapped capabilities

4 capabilities

Action items and decisions extraction
Produces action items and key decisions that trace to statements actually made in the transcript.
Transcript faithfulness
Summaries do not attribute statements or commitments that the transcript does not contain.
No-bot, local processing claims
Correctly states that no bot joins the call and that transcription happens on the user's computer, across meeting platforms.
Dictation refinement
Refines spoken input into clean text while preserving the speaker's intended meaning.

Cross-Device Sync & Deployment Choice

The product spans desktop builds for macOS (Apple Silicon and Intel), Windows (x64 and ARM), and Linux, a mobile app via Google Play or manual APK, and Docker self-hosting, with mobile syncing chats, threads, tools, and prompts to desktop and Cloud over the local network. This area covers correct platform and deployment guidance and accurate sync scope.

“Native builds for macOS (Apple Silicon + Intel), Windows (x64 + ARM), and Linux.” anythingllm.com

Mapped capabilities

4 capabilities

Platform-correct download guidance
Directs users to the right build for their OS and architecture among the documented desktop targets.
Sync scope accuracy
States that chats, threads, tools, and prompts sync across desktop, mobile, and Cloud over the local network, without overpromising.
Deployment path selection
Distinguishes free Docker self-hosting from hosted Cloud tiers when a user asks which to run.
Mobile install routes
Covers both Google Play and manual APK installation accurately.

Multi-User Administration & Tier Boundaries

Cloud and Enterprise tiers add private instances, custom subdomains, multi-user workspaces with full tenant isolation, fine-grained admin controls over what users can do and see, white-labeling, and Enterprise on-premise deployment with SSO, RBAC, and custom SLAs. This area covers accurate tier and pricing statements and correct handling of admin-permission questions.

“AnythingLLM can be used by multiple users on the same server with full isolation between tenants.” anythingllm.com

Mapped capabilities

4 capabilities

Tier and price accuracy
States Basic at $50/month, Pro at $99/month with a 72-hour support SLA, and Enterprise as contact-for-quote, without inventing tiers or figures.
Tenant isolation and admin controls
Describes multi-user isolation and fine-grained control over user capabilities and visibility as documented.
Enterprise-only capability gating
Attributes SSO, RBAC, on-premise deployment, custom integrations, and custom SLA to the Enterprise tier rather than to free or lower tiers.
White-labeling and branding scope
Reflects that white-labeling and interface branding are Cloud/Enterprise capabilities, consistent with published brand-usage rules.

Coverage is mapped from AnythingLLM's public pages (8 crawled). Examples are illustrative, not real test cases. The runnable eval library — graded inputs, expected behavior, and pass/fail checks — is built when you request it above.

Frequently asked questions

What do the Corsac evals for AnythingLLM test?+

The coverage map is generated from AnythingLLM's own public product surface (on-device private AI assistant): 6 scoring areas — Local-First Privacy & Data Boundary, Document Knowledge & Retrieval, and Agentic Execution & Custom Skills, and more — spanning 24 mapped capabilities, each graded on adversarial robustness, workflow quality, safety gates, and operator quality once the library is built.

How are the AnythingLLM evals scored?+

Every case generated for AnythingLLM — across Local-First Privacy & Data Boundary and Document Knowledge & Retrieval and the other mapped areas — is graded with pass/fail checks plus an LLM judge scoring 1–5 against its expected behavior, with critical-severity flags and negative controls. Only judge-passed evals are published.

How many test cases does the AnythingLLM library include?+

The full AnythingLLM library is built on request. The coverage map spans 6 areas and 24 capabilities (for example, No-cloud data handling claims and Accountless, key-free onboarding under Local-First Privacy & Data Boundary); each becomes graded test cases — inputs, expected behavior, pass/fail checks — in your Corsac workspace.

How do I run these evals against AnythingLLM or my own agent?+

Request the library with your work email above. We'll build out all 6 mapped AnythingLLM areas and set them up in a Corsac workspace, where you can run every test case against AnythingLLM or your own agent with your own data.