Retrieval And Vector Stores

Integrator stores retrieval text as bare strings, losing source/url metadata needed for citations and filtering.

Wrap chunks as Document(page_content=..., metadata={'source':..., 'page':...}). Metadata carries provenance for citations and enables metadata-filtered retrieval. Preserve it through splitting and embedding so it survives to the answer.

Integrator splits documents into 4000-character chunks with zero overlap and gets answers that miss facts spanning chunk boundaries.

Use RecursiveCharacterTextSplitter with a chunk_size matched to the embedding/model context and a non-zero chunk_overlap so boundary-spanning facts are retrievable. Tune sizes to the corpus; do not assume one size fits all documents.

A RAG chain answers from the model's parametric knowledge even when the retrieved context does not support the claim, and cites no source.

Prompt the model to answer ONLY from retrieved context and to say it does not know when context is insufficient; pass Document metadata through so the answer can cite sources. Ungrounded claims must be avoided, not silently filled from parametric memory.