多语言提示压缩中,非英语的压缩效果远差于英语,且模型训练数据决定性能差距。
Lost in Compression: A Controlled Cross-Lingual Audit of Extractive Prompt Compressors

- 用平行语料和匹配预算测试十种语言的压缩器,控制变量验证效果
- 中文等非英语语言在压缩后保留信息极少,极端情况下接近无上下文效果
- 多语言训练的压缩器能消除语言差距,翻译后再压缩更高效
提取式提示压缩可降低大模型推理成本,但现有方法主要针对英文。其他语言本身存在1.3-1.8倍的令牌溢价。本文使用涵盖五种文字的十种语言平行数据,在11个目标模型(来自10家厂商)上,对4个学习型压缩器与4个确定性基线进行审计,共执行超25万次评估。结果显示:跨语言压缩差距真实存在,且高度依赖率——在0.33保留率下,英语保留57-62%上下文利用率,而立陶宛语仅10-24%,中文几乎为零;该差距源于压缩器训练数据而非架构,三个仅用英语训练的压缩器均出现此现象,而多语言训练的XProvence v1无此问题,其v2版本在激进阈值下可清除92%中文上下文且无预警;在长上下文任务中,部分非英语语言的压缩结果甚至低于无上下文表现。翻译后再压缩的管道在五种语言中三种优于原生压缩,且成本减半。代码、压缩结果及输出均已开源。
原文摘要 · Abstract (English)
Extractive prompt compression promises to cut LLM inference costs by removing low-information tokens, and learned compressors such as LLMLingua-2 report strong results on English benchmarks. Most other languages already pay a token premium: the same content costs 1.3-1.8x more tokens than in English. We ask whether compression closes or widens this gap. Using fully parallel data in ten languages spanning five scripts, with controls budget-matched in the target model's tokenizer, we audit four learned compressors against four deterministic baselines, on eleven target models from ten vendors (over 250,000 evaluation calls). Three of the compressors are trained with English supervision (LLMLingua-2 XLM-R/mBERT; Kompress-v2 from the production Headroom stack); the fourth, XProvence, is trained multilingually. First, the transfer gap is real, replicates across target models and compressor backbones, and is strongly rate-dependent: at a 0.33 keep-rate English retains 57-62% of normalized context utilization while Lithuanian retains 10-24% and Chinese essentially none, despite Chinese having the smallest token premium. Second, the gap tracks compression supervision data, not architecture. All three English-trained compressors show it, deterministic methods show no comparable gap, and the multilingually trained XProvence v1 shows none. Its v2 release, retrained on translated data, empties 92% of Chinese contexts at its aggressive threshold without any warning. Third, in a harder long-context setting, aggressive learned compression drives compressed contexts to or below no-context utility in three of five non-English languages. A translate-then-compress pipeline matches or beats native compression at roughly half the token cost in three of five tested languages. We release all code, compressions, and model outputs. Safe compression budgets are much smaller outside English.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。