用自适应语义树压缩长文本，兼顾精度与效率。
AdmTree: Compressing Lengthy Context with Adaptive Semantic Trees
- 根据信息密度动态分段，用摘要令牌构建语义二叉树
- 在长文本中保持高语义保真度，避免位置偏差和信息损失
- 轻量级结构，仅需少量新参数，适合部署到大模型
自注意力的二次复杂度限制了大语言模型处理长上下文的能力，而上下文压缩可缓解此计算瓶颈并保留关键语义信息。现有方法存在不足：显式压缩易丢失局部细节，隐式压缩则可能产生位置偏差、信息退化或无法捕捉长程语义依赖。本文提出AdmTree，一种以高语义保真度为核心目标的自适应层级压缩框架。该框架基于信息密度动态分割输入，使用概要令牌（gist tokens）将不同长度的片段作为语义二叉树的叶节点。结合轻量级聚合机制与冻结的主干大模型（显著减少新增可训练参数），实现高效的上下文层次抽象。通过同时保留细粒度细节与全局语义一致性，有效缓解位置偏差，并动态适应内容变化，显著提升长上下文的语义保留能力。
原文摘要 · Abstract (English)
The quadratic complexity of self-attention constrains Large Language Models (LLMs) in processing long contexts, a capability essential for many advanced applications. Context compression aims to alleviate this computational bottleneck while retaining critical semantic information. However, existing approaches often fall short: explicit methods may compromise local detail, whereas implicit methods can suffer from positional biases, information degradation, or an inability to capture long-range semantic dependencies. We propose AdmTree, a novel framework for adaptive, hierarchical context compression with a central focus on preserving high semantic fidelity while maintaining efficiency. AdmTree dynamically segments input based on information density, utilizing gist tokens to summarize variable-length segments as the leaves of a semantic binary tree. This structure, together with a lightweight aggregation mechanism and a frozen backbone LLM (thereby minimizing new trainable parameters), enables efficient hierarchical abstraction of the context. By preserving fine-grained details alongside global semantic coherence, mitigating positional bias, and dynamically adapting to content, AdmTree robustly retains the semantic information of long contexts.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。