Quantization And Optimization

To cut memory, agent enables scalar (int8) quantization with quantization_config={scalar:{type:'int8', always_ram:true}}.

Enable scalar int8 quantization to shrink vector memory ~4x; place quantized vectors always_ram for fast first-stage search while keeping originals on disk for rescore. Validate recall after enabling — quantization trades a small accuracy loss for memory/speed. [REQUIRES-VERIFICATION] on exact reca…

Agent considers product quantization (PQ) for stronger compression than scalar but is unsure of the cost.

Product quantization compresses more than scalar int8 but costs more recall and more CPU at query time; it is a deliberate memory-vs-accuracy-vs-latency choice. Benchmark PQ vs scalar on your data with rescore enabled before committing; do not pick PQ purely for the compression ratio.

With quantization enabled, agent sets search params.quantization={rescore:true, oversampling:2.0} to recover recall.

At query time, oversampling fetches more candidates from the quantized index than limit, then rescore re-ranks them against the original (un-quantized) vectors so final top-k recall approaches un-quantized quality. Tune oversampling vs latency; rescore requires the originals to be available.