通过分层残差量化，让大模型向量搜索告别慢速存储读取。
FaTRQ: Tiered Residual Quantization for LLM Vector Search in Far-Memory-Aware ANNS Systems
- 用分层残差编码压缩向量，仅从远端内存流式读取小数据
- 查询延迟降低9倍，存储效率提升2.4倍，优于现有最先进系统
- 适合需要高吞吐向量搜索的RAG应用，如大模型问答与检索
近似最近邻搜索（ANNS）是检索增强生成（RAG）的核心技术，可从海量向量数据库中快速定位相关高维嵌入。现代ANNS引擎通过预构建索引和压缩向量量化表示加速查询，但仍依赖从慢速存储（如SSD）读取全精度向量的二次精炼阶段。对于现代文本与多模态嵌入，此类读取已成为整体延迟的主要瓶颈。本文提出FaTRQ，一种面向远端内存的精炼系统，采用分层内存架构，消除从存储中读取完整向量的需求。其引入渐进式距离估计器，利用从远端内存流式传输的紧凑残差值逐步优化粗略得分。一旦候选结果被证明不在前k名内，即停止精炼。为此，我们提出分层残差量化方法，将残差编码为三值形式，高效存储于远端内存。在CXL Type-2设备中部署专用加速器，实现低延迟本地精炼。实验表明，FaTRQ在存储效率上提升2.4倍，在吞吐量上最高达9倍，优于当前最先进的GPU ANNS系统。
原文摘要 · Abstract (English)
Approximate Nearest-Neighbor Search (ANNS) is a key technique in retrieval-augmented generation (RAG), enabling rapid identification of the most relevant high-dimensional embeddings from massive vector databases. Modern ANNS engines accelerate this process using prebuilt indexes and store compressed vector-quantized representations in fast memory. However, they still rely on a costly second-pass refinement stage that reads full-precision vectors from slower storage like SSDs. For modern text and multimodal embeddings, these reads now dominate the latency of the entire query. We propose FaTRQ, a far-memory-aware refinement system using tiered memory that eliminates the need to fetch full vectors from storage. It introduces a progressive distance estimator that refines coarse scores using compact residuals streamed from far memory. Refinement stops early once a candidate is provably outside the top-k. To support this, we propose tiered residual quantization, which encodes residuals as ternary values stored efficiently in far memory. A custom accelerator is deployed in a CXL Type-2 device to perform low-latency refinement locally. Together, FaTRQ improves the storage efficiency by 2.4$\times$ and improves the throughput by up to 9$ \times$ than SOTA GPU ANNS system.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。