Retrieval And Vector Stores

Integrator stores retrieval text as bare strings, losing source/url metadata needed for citations and filtering.

Wrap chunks as Document(page_content=..., metadata={'source':..., 'page':...}). Metadata carries provenance for citations and enables metadata-filtered retrieval. Preserve it through splitting and embedding so it survives to the answer.

Integrator splits documents into 4000-character chunks with zero overlap and gets answers that miss facts spanning chunk boundaries.

Use RecursiveCharacterTextSplitter with a chunk_size matched to the embedding/model context and a non-zero chunk_overlap so boundary-spanning facts are retrievable. Tune sizes to the corpus; do not assume one size fits all documents.

Integrator indexes documents with one embedding model and queries with a different embedding model, producing meaningless similarity scores.

Use the SAME Embeddings model (same provider/model/dimension) for indexing and querying. Embeddings exposes embed_documents and embed_query; mixing models or dimensions makes the vector space incomparable. Re-embed the corpus if the model changes.