Documents Nodes And Ingestion

An IngestionPipeline re-runs nightly over a folder of contracts. The loader assigns a fresh random Document.id_ on every run instead of a stable doc_id derived from the source file.

Set a stable Document.id_ (e.g. derived from the file path or a content/source key) so the pipeline's docstore can detect unchanged documents and dedup them. With a docstore attached, unchanged docs are skipped and changed docs are upserted — random ids defeat dedup and re-embed everything every ru…

Each Document carries metadata including a large 'raw_html' blob and an internal 'tenant_id'. The integrator wants tenant_id usable for filtering but never sent to the LLM or embedded, and raw_html excluded from both.

Populate Node.excluded_llm_metadata_keys and excluded_embed_metadata_keys so raw_html and tenant_id are stripped from the text seen by the LLM and the embedding model, while remaining available as metadata for filters. Verify via node.get_content(metadata_mode=...) which keys leak into each context.

A SentenceSplitter is configured with chunk_size=4096 tokens and chunk_overlap=0 for an embedding model whose max input is 512 tokens.

Choose chunk_size compatible with the embedding model's max sequence length so nodes are not silently truncated at embed time. Set a sensible chunk_overlap to preserve cross-boundary context. Verify node token lengths against the embedder's limit before indexing.