提出简单高效基线，显著提升上下文压缩效果
No Mean Feat: Simple, Strong Baselines for Context Compression
- 用均值池化和双向压缩令牌实现高效上下文压缩
- 在小于8000词元的上下文中性能超越主流方法
- 适合关注RAG与推理加速的研究者参考
上下文压缩通过将长输入替换为预计算的短表示来降低Transformer推理开销，对检索增强生成（RAG）有显著价值，正受到越来越多关注。然而，由于评估不一致和基线模糊，进展难以衡量。本文设计了标准化、可复现的评估套件BenchPress，支持跨模型规模、数据集、压缩比及短（<1K词元）到中等（<8K词元）上下文的基准测试。尽管该套件适用于任何压缩范式，我们的基线聚焦软压缩。我们提出了两种简单但高性能的基线：均值池化和双向压缩令牌变体，其表现显著优于广泛使用的因果压缩令牌方法。结果表明，计算压缩表示时使用双向注意力更有效，且简单的池化操作本身具有强表达能力。
原文摘要 · Abstract (English)
Context compression reduces Transformer inference costs by replacing lengthy inputs with shorter pre-computed representations. It carries significant benefits for retrieval-augmented generation (RAG) and has attracted growing research attention. However, progress remains difficult to measure due to inconsistent evaluations and baselines. We design a standard, easy-to-reproduce evaluation suite for context compression, BenchPress, along with simple, high-performance baselines for English reading comprehension. BenchPress supports benchmarking across model scales, datasets, compression ratios, and short ($<$1K tokens) to mid-range ($<$8K tokens) contexts. While the suite is applicable to any compression paradigm, our baselines target soft context compression. We establish two simple baselines that strongly outperform the widely used causal compression-token approach: mean pooling and a bidirectional compression-token variant. Our results show the benefit of bidirectional attention when computing compressed representations, and that simple pooling is an expressive compression operator.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。