让AI持续学习多模态数据，还能保证检索结果稳定可靠。
LUMA-RAG: Lifelong Multimodal Agents with Provably Stable Streaming Alignment
- 用分层内存动态管理多模态数据，节省存储成本。
- 跨模态对齐精度达94%（Recall@10），音频到图像检索稳定无误。
- 适合需要长期更新、高稳定性的多模态应用开发。
检索增强生成（RAG）已成为将大模型输出与可验证证据结合的主流范式。然而，随着AI智能体从静态知识库转向持续的多模态数据流（文本、图像、视频、音频），两个关键挑战浮现：在不产生高昂重索引成本的前提下保持索引新鲜度，以及在异构嵌入空间中维持跨模态语义一致性。本文提出LUMA-RAG，一种终身多模态智能体架构，包含三项核心创新：(i) 流式多级记忆系统，在严格内存预算下，动态将嵌入向量从热区HNSW层级溢出至压缩的IVFPQ层级；(ii) 流式CLAP→CLIP对齐桥，通过增量正交Procrustes更新维持跨模态一致性；(iii) 稳定性感知检索遥测，联合约束对齐漂移与量化误差，实现Safe@k保障。实验表明，该系统在文本到图像检索中表现稳健（Recall@10 = 0.94），在产品量化卸载下性能平稳下降，并实现了音频到图像排序的可证明稳定性（Safe@1 = 1.0），确立了LUMA-RAG作为生产级多模态RAG系统的实用性。
原文摘要 · Abstract (English)
Retrieval-Augmented Generation (RAG) has emerged as the dominant paradigm for grounding large language model outputs in verifiable evidence. However, as modern AI agents transition from static knowledge bases to continuous multimodal streams encompassing text, images, video, and audio, two critical challenges arise: maintaining index freshness without prohibitive re-indexing costs, and preserving cross-modal semantic consistency across heterogeneous embedding spaces. We present LUMA-RAG, a lifelong multimodal agent architecture featuring three key innovations: (i) a streaming, multi-tier memory system that dynamically spills embeddings from a hot HNSW tier to a compressed IVFPQ tier under strict memory budgets; (ii) a streaming CLAP->CLIP alignment bridge that maintains cross-modal consistency through incremental orthogonal Procrustes updates; and (iii) stability-aware retrieval telemetry providing Safe@k guarantees by jointly bounding alignment drift and quantization error. Experiments demonstrate robust text-to-image retrieval (Recall@10 = 0.94), graceful performance degradation under product quantization offloading, and provably stable audio-to-image rankings (Safe@1 = 1.0), establishing LUMA-RAG as a practical framework for production multimodal RAG systems.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。