arXiv:2603.23527cs.CL2026-03被引 2

压缩方法影响输出长度,不同数据集表现差异大。

Compression Method Matters: Benchmark-Dependent Output Dynamics in LLM Prompt Compression

  • 提出指令存活概率Psi,衡量压缩后关键提示是否保留。
  • 在MBPP上输出膨胀56倍,人类评估仅5倍,结果受数据集影响显著。
  • 建议跨数据集测试,避免单一评估误导压缩安全判断。

提示压缩常以输入词元减少来评估,但实际部署影响取决于压缩对输出长度和总推理成本的影响。我们对三种基准、多个供应商进行了大规模的可控复现与扩展研究,共覆盖5,400次API调用。为解释先前矛盾结论,我们形式化定义了指令存活概率(Psi),一种捕捉压缩后任务关键提示段是否保留的结构性指标。结果显示强烈的数据集效应:在压缩率r=0.3时,DeepSeek在MBPP上出现严重输出膨胀(56倍,Psi≈0.15),而在HumanEval上仅5倍(Psi≈0.72);GPT-4o-mini则在各基准上相对稳定。该发现揭示提示结构而非模型身份才是主要调节因素。我们引入压缩鲁棒性指数(CRI)用于跨基准评估,表明单基准评估可能产生误导性结论。结合租用RunPod GPU的NVML直接测量,我们指出词元节省可能夸大能耗节约。这些结果推动采用多样化基准与结构感知的压缩策略,实现更可靠、节能的LLM部署。

原文摘要 · Abstract (English)

Prompt compression is often evaluated by input-token reduction, but its real deployment impact depends on how compression changes output length and total inference cost. We present a controlled replication and extension study of benchmark-dependent output dynamics under aggressive compression, covering 5,400 API calls across three benchmarks and multiple providers. To explain conflicting prior observations, we formalize instruction survival probability (Psi), a structural metric that captures whether task-critical prompt segments remain after truncation. Results show a strong benchmark effect: under r=0.3, DeepSeek exhibits severe output expansion on MBPP (56x, Psi approx 0.15) but substantially lower expansion on HumanEval (5x, Psi approx 0.72), while GPT-4o-mini is comparatively stable across benchmarks. This reconciles the apparent discrepancy between previously reported extreme explosion and lower replication effects by identifying prompt structure, not provider identity alone, as the primary moderator. We introduce the Compression Robustness Index (CRI) for cross-benchmark evaluation and show that single-benchmark assessments can produce misleading conclusions about compression safety and efficiency. To contextualize energy claims, we incorporate companion direct NVML measurements from rented RunPod GPUs and show that token savings can overstate joule savings. These findings motivate benchmark-diverse testing and structure-aware compression policies for reliable, energy-conscious LLM deployment.

提示压缩模型评估能效分析

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。