实时更新向量知识库，兼顾查询速度与版本追溯。
LiveVectorLake: A Real-Time Versioned Knowledge Base Architecture for Streaming Vector Updates and Temporal Retrieval
- 用SHA-256哈希实现块级内容同步，无需外部状态追踪。
- 热冷分层存储：当前数据低延迟检索，历史版本低成本保存。
- 支持任意时间点查询，满足合规审计需求，适合生产级RAG系统。
现代检索增强生成（RAG）系统面临核心架构矛盾：向量索引优化查询延迟但难以处理持续知识更新，而数据湖虽擅长版本管理却带来查询延迟。我们提出LiveVectorLake，一种双层时序知识库架构，可在保持完整版本历史以满足合规、审计和任意时间点检索的同时，实现当前知识的实时语义搜索。系统引入三项核心设计：(1) 基于SHA-256哈希的可内容寻址块级同步，实现确定性变更检测，无需外部状态追踪；(2) 双层存储结构，将热层向量索引（Milvus + HNSW）与冷层列式版本化存储（Delta Lake + Parquet）分离，独立优化查询延迟与存储成本；(3) 时序查询路由机制，通过增量版本控制实现跨层级的ACID一致性点时间检索。在包含100个文档、经五个时间点版本化的数据集上评估显示：(i) 更新时仅需重处理10%-15%内容，远低于全量重索引的100%；(ii) 当前知识检索延迟低于100ms；(iii) 历史版本查询延迟低于2秒；(iv) 通过热/冷分层显著降低存储成本（仅当前块驻留昂贵索引中）。该方案支持需同时优化查询性能、更新效率与监管合规的生产级RAG部署。代码与资源：[https://github.com/praj-tarun/LiveVectorLake]
原文摘要 · Abstract (English)
Modern Retrieval-Augmented Generation (RAG) systems struggle with a fundamental architectural tension: vector indices are optimized for query latency but poorly handle continuous knowledge updates, while data lakes excel at versioning but introduce query latency penalties. We introduce LiveVectorLake, a dual-tier temporal knowledge base architecture that enables real-time semantic search on current knowledge while maintaining complete version history for compliance, auditability, and point-in-time retrieval. The system introduces three core architectural contributions: (1) Content-addressable chunk-level synchronization using SHA-256 hashing for deterministic change detection without external state tracking; (2) Dual-tier storage separating hot-tier vector indices (Milvus with HNSW) from cold-tier columnar versioning (Delta Lake with Parquet), optimizing query latency and storage cost independently; (3) Temporal query routing enabling point-in-time knowledge retrieval via delta-versioning with ACID consistency across tiers. Evaluation on a 100-document corpus versioned across five time points demonstrates: (i) 10-15% re-processing of content during updates compared to 100% for full re-indexing; (ii) sub-100ms retrieval latency on current knowledge; (iii) sub-2s latency for temporal queries across version history; and (iv) storage cost optimization through hot/cold tier separation (only current chunks in expensive vector indices). The approach enables production RAG deployments requiring simultaneous optimization for query performance, update efficiency, and regulatory compliance. Code and resources: [https://github.com/praj-tarun/LiveVectorLake]
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。