Documents Nodes And Ingestion

An IngestionPipeline re-runs nightly over a folder of contracts. The loader assigns a fresh random Document.id_ on every run instead of a stable doc_id derived from the source file.

Set a stable Document.id_ (e.g. derived from the file path or a content/source key) so the pipeline's docstore can detect unchanged documents and dedup them. With a docstore attached, unchanged docs are skipped and changed docs are upserted — random ids defeat dedup and re-embed everything every ru…

After custom splitting, the integrator builds TextNodes manually but does not set NodeRelationship.SOURCE / PREVIOUS / NEXT. Later, prev/next-window expansion retrieval returns nothing.

Preserve node relationships (SOURCE back to the Document, PREVIOUS/NEXT between adjacent chunks) — either by using a built-in node parser or by setting relationships explicitly. Postprocessors like PrevNextNodePostprocessor and auto-merging retrieval depend on these links.

Each Document carries metadata including a large 'raw_html' blob and an internal 'tenant_id'. The integrator wants tenant_id usable for filtering but never sent to the LLM or embedded, and raw_html excluded from both.

Populate Node.excluded_llm_metadata_keys and excluded_embed_metadata_keys so raw_html and tenant_id are stripped from the text seen by the LLM and the embedding model, while remaining available as metadata for filters. Verify via node.get_content(metadata_mode=...) which keys leak into each context.