Eval directory

Evals for Agno

Eval coverage for Agno, mapped from its public product surface.

About Agno

Agno is an open-source Python framework and high-performance runtime for building multi-agent systems. Its AgentOS product provides a Control Plane for running, chatting with, and monitoring agents, teams, and workflows, with session metrics, knowledge/memory management, and evaluations. It is model-agnostic and offered as a free open-source tier plus paid Pro and Enterprise plans with self-hosting, SSO/RBAC, and support SLAs.

Industry

multi-agent AI framework and runtime

Website

www.agno.com

Use the eval library for Agno

We'll build out the full library — runnable test cases with inputs, expected behavior, and pass/fail checks — in your Corsac workspace.

All evals →

More AI eval libraries

Coverage map

What would you measure for Agno?

6 scoring areas · 24 capabilities mapped · grounded in 8 cited pages

Every eval set is graded on

Adversarial robustness
Workflow quality
Safety gates
Operator quality

Pass/Fail + LLM judge 1–5 · critical severity flags · negative controls

Agent, Team & Workflow Authoring

Building multi-agent systems in plain Python with Agno's framework primitives, without graphs or chains.

“Agno helps you run a private and secure agent platform.” www.agno.com

Mapped capabilities

4 capabilities

Agent construction in pure Python
Instantiating an Agent with instructions, tools, and output options as shown in documented snippets.
Teams of agents
Composing multiple agents into a team as a supported framework primitive.
Workflows
Defining workflows that AgentOS can run, chat with, and monitor.
Model-agnostic provider selection
Swapping model/provider/modality without restructuring agent code, per Agno's agnostic design claim.

AgentOS Runtime & Control Plane

Running agents, teams, and workflows on the AgentOS runtime and driving them from the Control Plane.

“Control Plane for live AgentOS 1 live connection 4 total seats included” www.agno.com

Mapped capabilities

4 capabilities

Running agent systems on AgentOS
Executing agents, teams, and workflows on the runtime/API server.
Chatting with agents, teams, and workflows
Interactive chat surface exposed by the Control Plane.
Local vs. live AgentOS connections
Control Plane for local AgentOS on the free tier versus live connections on Pro.
Self-hosted Control Plane
Enterprise self-hosting of the Control Plane for private/secure deployment.

Monitoring, Metrics & Evaluations

Observing agent behavior over time and running system evaluations from AgentOS.

“unlimited monitoring unlimited retention unlimited knowledge unlimited memories unlimited chats” www.agno.com

Mapped capabilities

4 capabilities

Session monitoring
Reviewing agent, team, and workflow sessions in the Control Plane.
Session metrics
Metrics surfaced per session for running systems.
System evaluations
The evaluations capability listed as part of AgentOS.
Monitoring and retention limits by plan
Unlimited monitoring and retention on Pro versus the free tier's local scope.

Knowledge & Memory Management

Backing agents with knowledge bases, vector search, and persistent memories.

“Uncompromising Performance: Blazing-fast Agents with a minimal memory footprint.” www.agno.com

Mapped capabilities

4 capabilities

Knowledge base setup
Attaching a Knowledge object to agents via agno.knowledge.knowledge.
Vector database backends
Supported vector DBs such as agno.vectordb.opensearch.
Vector, keyword, and hybrid search
Choosing semantic, lexical, or combined retrieval, sync or async.
Memories
Memory management surfaced in AgentOS, with unlimited memories on Pro.

Illustrative example

Input: Can I back an Agno knowledge base with OpenSearch and get both semantic and keyword matching in one query?
Expected behavior: Confirms OpenSearch is a supported vector database via agno.vectordb.opensearch, and notes it handles vector, keyword, and hybrid search in both sync and async variants, so hybrid combines the two in one query.

Tools & Integrations

Extending agents with Agno toolkits and configuring their credentials and outputs.

Mapped capabilities

4 capabilities

Toolkit attachment
Passing toolkits (e.g. SmallestTools) into an Agent's tools list.
Credential and environment configuration
Provider API keys such as SMALLEST_API_KEY with no extra install beyond agno.
Tool call selection
Choosing the right documented tool function for a request, e.g. text_to_speech vs. get_voices.
Artifact handling from tool output
Saving to target_directory or reading bytes from response.audio for a later step.

Plans, Access & Governance

Tier boundaries, seat/connection economics, and enterprise access controls.

Mapped capabilities

4 capabilities

Tier boundaries
What Free, Pro ($150/mo), and Enterprise each include.
Seats and live connections
4 seats and 1 live connection included; $30/mo per seat and $95/mo per live connection add-ons.
SSO and RBAC
Custom SSO and RBAC offered on Enterprise.
Support channels and SLA
Community/Discord and docs on Free versus dedicated Slack, technical lead, and support SLA on Enterprise.

Illustrative example

Input: We want Agno Pro for 7 people and need 3 live AgentOS connections. What does that cost per month?
Expected behavior: States $430/mo total, and shows the arithmetic: the $150/mo Pro base includes 4 seats and 1 live connection, so 3 extra seats at $30 and 2 extra live connections at $95 are added.

Coverage is mapped from Agno's public pages (8 crawled). Examples are illustrative, not real test cases. The runnable eval library — graded inputs, expected behavior, and pass/fail checks — is built when you request it above.

Frequently asked questions

What do the Corsac evals for Agno test?+

The coverage map is generated from Agno's own public product surface (multi-agent AI framework and runtime): 6 scoring areas — Agent, Team & Workflow Authoring, AgentOS Runtime & Control Plane, and Monitoring, Metrics & Evaluations, and more — spanning 24 mapped capabilities, each graded on adversarial robustness, workflow quality, safety gates, and operator quality once the library is built.

How are the Agno evals scored?+

Every case generated for Agno — across Agent, Team & Workflow Authoring and AgentOS Runtime & Control Plane and the other mapped areas — is graded with pass/fail checks plus an LLM judge scoring 1–5 against its expected behavior, with critical-severity flags and negative controls. Only judge-passed evals are published.

How many test cases does the Agno library include?+

The full Agno library is built on request. The coverage map spans 6 areas and 24 capabilities (for example, Agent construction in pure Python and Teams of agents under Agent, Team & Workflow Authoring); each becomes graded test cases — inputs, expected behavior, pass/fail checks — in your Corsac workspace.

How do I run these evals against Agno or my own agent?+

Request the library with your work email above. We'll build out all 6 mapped Agno areas and set them up in a Corsac workspace, where you can run every test case against Agno or your own agent with your own data.