不训练模型,用图结构高效压缩长文本,保持语义连贯性。
From Similarity to Structure: Training-free LLM Context Compression with Hybrid Graph Priors

- 构建混合图结构:结合语义相似与顺序关系,捕捉句子间深层关联。
- 在四数据集上超越主流方法,长文档压缩效果更优,性能接近抽象型摘要。
- 无需训练、可适配任意模型,适合对推理成本敏感的场景。
长上下文大语言模型运行成本高,且难以可靠处理超长输入,因此上下文压缩成为关键组件。现有方法多依赖训练过的压缩器、密集检索式选择或启发式截断,常难以在严格令牌预算下同时保留任务相关性、主题覆盖和跨句连贯性。为此,我们提出一种无需训练、模型无关的压缩框架,通过结构化图先验引导句子选择。该方法构建稀疏混合句子图,融合互k-NN语义边与短程顺序边,利用聚类提取主题骨架,并基于任务相关性、簇代表性、桥接中心性及环覆盖线索设计可解释评分函数。通过带预算的贪心选择并抑制冗余,最终输出保持原文顺序的可读压缩上下文。在四个数据集上的实验表明,该方法在性能上媲美强基准,尤其在长文档基准上表现更优。
原文摘要 · Abstract (English)
Long-context large language models remain computationally expensive to run and often fail to reliably process very long inputs, which makes context compression an important component of many systems. Existing compression approaches typically rely on trained compressors, dense retrieval-style selection, or heuristic trimming, and they often struggle to jointly preserve task relevance, topic coverage, and cross-sentence coherence under a strict token budget. To address this, we propose a training-free and model-agnostic compression framework that selects a compact set of sentences guided by structural graph priors. Our method constructs a sparse hybrid sentence graph that combines mutual k-NN semantic edges with short-range sequential edges, extracts a topic skeleton via clustering, and ranks sentences using an interpretable score that integrates task relevance, cluster representativeness, bridge centrality, and a cycle coverage cue. A budgeted greedy selection with redundancy suppression then produces a readable compressed context in original order. Experimental results on four datasets show that our approach is competitive with strong extractive and abstractive baselines, demonstrating larger gains on long-document benchmarks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。