固定压缩会掩盖模型真实性能，误导评估结果
Fixed RAG Compression Collapses Measured Reader Scaling
- 用固定压缩方法评估模型时，压缩本身会改变读者表现
- 压缩提升弱模型准确率，却压制强模型潜力，导致排名反转
- 适合评估压缩效果的论文作者使用该工具验证结果可靠性
检索增强生成（RAG）压缩研究常在1到3个阅读器上评估压缩器，并假设压缩后的证据层对评估无影响。我们发现这一假设错误：固定压缩虽可提升平均准确率，但会隐藏阅读器升级效果并颠倒模型排名。在20个阅读器和10种领域-方法组合下，覆盖四个问答基准和一个摘要基准，压缩增益随阅读器基线准确率上升而下降（10组中有9组显著，p < 0.05）。通用摘要在LongMemEval-S上翻转了31%的模型配对排名，而固定的HotpotQA压缩器掩盖了从Qwen 7B到GPT-4.1-mini的80%原始性能提升。两种相反机制解释此悖论：压缩通过去除噪声拯救弱阅读器，同时因丢失细节损害强阅读器。该现象出现在结构化编译、通用摘要、三种训练压缩器族、查询聚焦摘要及对九篇已发表压缩论文的外部审计中。我们发布ragscale工具包，基于17.7万条行级压缩转换数据，使任意压缩论文可在一天内用三个阅读器完成读者扩展性审计。
原文摘要 · Abstract (English)
Retrieval-Augmented Generation (RAG) compression papers often evaluate a compressor on one to three readers and treat the compressed evidence layer as evaluation-neutral. We show this assumption is false: fixed compression can raise average accuracy while hiding reader upgrades and reversing model rankings. Across 20 readers and ten domain-method settings over four QA benchmarks and one summarization benchmark, compression gain decreases with reader baseline (nine of ten settings significant, p < 0.05). Generic summarization flips 31% of pairwise model rankings on LongMemEval-S, and a fixed HotpotQA compressor hides 80% of the raw upgrade from Qwen 7B to GPT-4.1-mini. Two opposing forces explain this paradox: compression rescues weak readers by removing noise they cannot filter, and harms strong readers by dropping details they would have used. The pattern appears across structured compilation, generic summarization, three trained compressor families, query-focused summarization, and an external audit of nine published compression papers. We release ragscale, a toolkit built on 177,000 row-level compression transitions, so any compression paper can audit reader scaling with three readers in one day.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。