Chunking For Embeddings

The integrator chunks parsed output by a fixed 1,000-character window, slicing mid-sentence and mid-table, instead of using Reducto's layout-aware block boundaries.

Chunk on semantic/layout boundaries (section, paragraph, table) surfaced by the parse output rather than a blind character window that splits sentences, tables, and list items. Layout-aware chunks improve retrieval precision. Where a hard size cap is needed, split at the nearest block boundary, not…

A 200-row table is embedded as one giant chunk that blows the embedding model's context, or split row-by-row losing the header context.

Chunk large tables deliberately: keep the header with each row-group so a retrieved slice retains column semantics, and bound chunk size to the embedding model's context. Neither one-giant-chunk nor headerless-row-shards serves retrieval. Consider a row-group + header-prefix strategy.

Many documents share boilerplate (terms, disclaimers). The integrator embeds every near-identical boilerplate chunk, flooding retrieval with duplicates and crowding out distinctive content.

Deduplicate near-identical boilerplate chunks (exact-hash + near-dup detection) before or during indexing so retrieval surfaces distinctive content, not the same disclaimer 400 times. Keep one canonical copy with references. Measure the duplicate rate in the index.