LlamaIndex
For LlamaIndexAI Platform

Indexes

LlamaIndex (+ LlamaCloud) · LlamaIndex

RAG / Data Framework — LlamaIndex

Evaluates LlamaIndex's Indexes across 9 scenario-based test cases, each graded against an expected-behavior rubric by an LLM judge, from Corsac's RAG / Data Framework eval coverage.

About LlamaIndex

LlamaIndex is a data framework for building RAG and agent applications over private data — documents/nodes, indexes (VectorStoreIndex), retrievers and query engines, the IngestionPipeline, plus LlamaParse and LlamaCloud for managed document parsing and retrieval.

Employees

~50

Industry

RAG Framework

Headquarters

San Francisco, CA

Sample tests· showing 3 of 9

#InputExpected behaviorCheck
01

A VectorStoreIndex built with the default SimpleVectorStore is used in-process, then the service restarts and rebuilds the index from documents every boot, re-embedding the whole corpus.

Persist via storage_context.persist(persist_dir=...) (docstore + index_store + vector store) and reload with load_index_from_storage(StorageContext.from_defaults(persist_dir=...)). Do not re-run from_documents on every boot. For external vector stores, reconstruct the index from the existing store,…

Pass / FailAi Platformhigh
02

An integrator calls VectorStoreIndex.from_documents(docs) expecting it to write to their Qdrant collection, but never passed a StorageContext, so nodes land in the in-memory SimpleVectorStore.

Pass storage_context=StorageContext.from_defaults(vector_store=QdrantVectorStore(...)) (or build via VectorStoreIndex.from_vector_store(...)). Confirm vectors actually land in Qdrant — the default in-memory store silently 'works' until restart, masking the misconfiguration.

Pass / FailAi Platformcritical
03

New documents arrive hourly. The integrator inserts them directly into the external vector store via its native client, bypassing index.insert / insert_nodes, so the LlamaIndex docstore never learns about them.

Insert through the index (index.insert(document) or insert_nodes(nodes)) so the docstore, index_store, and vector store stay consistent. Bypassing LlamaIndex to write the vector store directly leaves refresh/delete and ref-doc tracking unable to manage those nodes.

Pass / FailAi Platformhigh

Unlock full benchmark

6 more test cases

Use this benchmark

How this eval is graded

Grade against expected.ideal_behavior and expected.rubric. Per-criterion pass requires mean >= 4.0 and no criterion below 3.

Rubric criteria

  • Llamaindex
  • Ai Platform
  • Indexes

Recommended for

LlamaIndex (+ LlamaCloud)LlamaIndex customers

Works with

Related evals

Frequently asked questions

What does the Indexes eval for LlamaIndex LlamaIndex (+ LlamaCloud) test?+

Evaluates LlamaIndex's Indexes across 9 scenario-based test cases, each graded against an expected-behavior rubric by an LLM judge, from Corsac's RAG / Data Framework eval coverage.

How is the Indexes eval scored?+

The judge rubric: Grade against expected.ideal_behavior and expected.rubric. Per-criterion pass requires mean >= 4.0 and no criterion below 3.

How many test cases does this eval pack include?+

The Indexes pack for LlamaIndex LlamaIndex (+ LlamaCloud) contains 9 test cases. 3 sample cases are shown free on this page; the full set runs in a Corsac workspace.

How do I run this eval?+

Sign up for Corsac, connect your model or agent endpoint, and run the Indexes pack as-is or after customizing thresholds. Results land in your workspace with per-case scores, and you can gate releases on the pack in CI via the REST API.

Run this eval in your workspace

Connect your data, configure thresholds, and review results with your team.