让嵌入向量能随上下文动态演化,提升长文本检索与智能体记忆效果。
EvoEmbedding: Evolvable Representations for Long-Context Retrieval and Agentic Memory

- 通过持续更新的隐式记忆,结合原始内容生成可演化的嵌入表示。
- 在多个长上下文检索任务上超越更大模型,支持10倍于训练窗口的上下文。
- 适合需要动态记忆的智能体系统,可直接提升现有RAG流程性能。
现有嵌入模型本质上是静态的:它们孤立地编码文本片段,忽略上下文和时间顺序。本文提出EvoEmbedding,一种专为长上下文场景设计的可演化嵌入模型,能够持续追踪信息状态。其核心机制是:在逐序处理输入时,维护一个持续更新的潜在记忆,并与原始内容联合生成可演化的嵌入表示。因此,同一查询在不同上下文中会生成不同的表征,实现动态检索,突破静态语义搜索的局限。为训练该能力,我们构建了EvoTrain-180K数据集,用于联合优化潜在记忆与检索性能。同时引入记忆队列防止表征坍缩,并采用段批处理技术缓解长度差异,使训练加速3.8倍。大量实验表明,该模型不仅优于更大型的专业模型(如Qwen3-Embedding-8B和KaLM-Embedding-Gemma3-12B),还能在上下文长达训练窗口10倍的情况下良好泛化至下游任务(如个性化)。尤其值得注意的是,将EvoEmbedding融入简单RAG流程后,性能超过专门的智能体记忆系统。
原文摘要 · Abstract (English)
Existing embedding models are inherently static: they encode text segments in isolation, ignoring their surrounding context and temporal order. This paper introduces EvoEmbedding, a novel embedding model that generates evolvable representations for retrieval. It is tailored for long-context scenarios, where information is dynamic, sequential, and requires continuous state tracking. Our design is simple: EvoEmbedding maintains a continuously updated latent memory as it sequentially processes inputs, and uses it alongside the raw content to jointly generate evolvable embeddings. Consequently, for the same query, our model adapts its representation to retrieve distinct targets based on the evolving context, going beyond static semantic search. To equip the model with this capability, we construct EvoTrain-180K, a diverse dataset for the joint optimization of latent memory and retrieval. Furthermore, we introduce a memory queue to prevent representation collapse during recurrent encoding, alongside segment-batching techniques that tackle significant length variance and accelerate training by 3.8$\times$. Extensive experiments show that our model not only outperforms larger-scale specialists (e.g., Qwen3-Embedding-8B and KaLM-Embedding-Gemma3-12B) across a range of long-context retrieval benchmarks, but also generalizes well to downstream tasks (e.g., personalization) with contexts 10$\times$ longer than its training window. Notably, EvoEmbedding seamlessly integrates into agentic workflows to boost performance. For instance, a naive RAG pipeline equipped with our model surpasses dedicated agentic memory systems. Project Page: https://clare-nie.github.io/EvoEmbedding/.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。