Embedding And Enrichment

An embedding transform is configured before chunking, so a 50-page document is embedded as a single oversized vector.

Place the embedding transform after partition + chunk so each chunk gets its own vector sized for the embedding model's context. Verify the embedded unit is a chunk (bounded by max_characters), not a whole document.

The workflow's embedding model emits 1536-dim vectors but the destination vector index was created for 768 dims; writes silently fail or are rejected.

Match the embedding transform's output dimension to the destination index's configured dimension. On a model change, re-create/migrate the index for the new dimension and re-embed; do not write mismatched-dimension vectors. [REQUIRES-VERIFICATION for a given model's exact dimension].

A summary-enrichment step overwrites element.text with the generated summary, so the original extracted text is lost before it reaches the index.

Attach enrichment outputs (summary, NER entities, image description) as additional metadata on the element, preserving the original element.text. Never replace the verbatim extracted text with model-generated content.