通过分组合并与语义对齐，压缩长文本上下文并提升模型效率。
GMSA: Enhancing Context Compression via Group Merging and Layer Semantic Alignment
- 分组合并实现更均匀的语义聚合，减少信息冗余。
- 在多个长文本问答与摘要任务上超越基线模型性能。
- 适用于需要高效处理长文本的场景，如文档摘要、对话系统。
大型语言模型（LLMs）在众多自然语言处理任务中表现优异。但在长上下文场景下，面临计算开销高和信息冗余的问题。为此，我们提出GMSA——一种编码器-解码器结构的上下文压缩框架，通过生成紧凑的软标记序列来支持下游任务。GMSA引入分组合并（Group Merging），实现更均匀的聚合，缓解自编码预训练中的语义主导问题；同时采用层间语义对齐（Layer Semantic Alignment, LSA），弥合高层抽象语义与底层输入语义之间的差距。我们先将GMSA作为自编码器进行预训练，再针对下游任务微调。实验表明，相较于现有软提示压缩范式，GMSA在上下文重建任务中表现更优，并在两种主干模型上的多个长文本问答与摘要基准测试中优于基线模型，同时保持低端到端延迟。
原文摘要 · Abstract (English)
Large Language Models (LLMs) have achieved remarkable performance across a wide range of Natural Language Processing (NLP) tasks. However, in long-context scenarios, they face two challenges: high computational cost and information redundancy. To address these challenges, we propose GMSA, an encoder-decoder context compression framework that generates a compact sequence of soft tokens for downstream tasks. GMSA introduces Group Merging to achieve more uniform aggregation, mitigating semantic dominance during autoencoder pretraining, and Layer Semantic Alignment (LSA) to bridge the semantic gap between high-level abstract semantics and low-level input semantics. We first pretrain GMSA as an autoencoder and then fine-tune it for downstream tasks. Experiments demonstrate that GMSA improves context reconstruction compared to existing soft prompt compression paradigm and outperforms baselines on multiple long-context question answering and summarization benchmarks across two backbone models, while maintaining low end-to-end latency.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。