通过分块摘要令牌提升长文本嵌入质量，解决信息回流瓶颈。
Hierarchical Token Prepending: Enhancing Information Flow in Decoder-based LLM Embeddings
- 分块插入摘要令牌，构建多路径信息回流机制
- 用均值池化替代最后令牌聚合，缓解信息过压缩
- 在11个检索数据集上稳定提升，适合长文档场景
大型语言模型生成强大文本嵌入，但其因果注意力机制限制了后续词元向早期词元的信息流动，降低表征质量。现有方法通过插入单一摘要词元解决此问题，但过度压缩信息，导致长文档性能下降。我们提出层级式词元前置（HTP），解决两大关键瓶颈：为缓解注意力层面的压缩，HTP将输入分块，并在后续块前插入块级摘要词元，建立多条反向信息流动路径；为应对读出层面的过挤压，采用均值池化替代最后词元池化，该选择经理论分析支持。HTP在11个检索数据集和30个通用嵌入基准上实现一致性能提升，尤其在长上下文场景中表现优异。作为一种简单且架构无关的方法，HTP可增强零样本与微调模型，提供更优长文档嵌入的可扩展路径。
原文摘要 · Abstract (English)
Large language models produce powerful text embeddings, but their causal attention mechanism restricts the flow of information from later to earlier tokens, degrading representation quality. While recent methods attempt to solve this by prepending a single summary token, they over-compress information, hence harming performance on long documents. We propose Hierarchical Token Prepending (HTP), a method that resolves two critical bottlenecks. To mitigate attention-level compression, HTP partitions the input into blocks and prepends block-level summary tokens to subsequent blocks, creating multiple pathways for backward information flow. To address readout-level over-squashing, we replace last-token pooling with mean-pooling, a choice supported by theoretical analysis. HTP achieves consistent performance gains across 11 retrieval datasets and 30 general embedding benchmarks, especially in long-context settings. As a simple, architecture-agnostic method, HTP enhances both zero-shot and finetuned models, offering a scalable route to superior long-document embeddings.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。