代码比数学更耐压缩，因模型对代码语法保留更严
The Perplexity Paradox: Why Code Compresses Better Than Math in LLM Prompts
- 分析每令牌困惑度，发现代码语法被保留而数学数值被丢弃
- 签名注入使通过率从5.3%提至39.3%，提升34个百分点
- 提出自适应压缩算法，成本降22%仍保96%质量，适合部署优化
在《Compress or Route?》（Johnson, 2026）中，我们发现代码生成能容忍高强度提示压缩（r ≥ 0.6），而链式思维推理则逐渐退化。该研究仅限于HumanEval（164个问题），未验证‘困惑度悖论’机制，也未提供自适应算法。本文填补三项空白：第一，在六项代码基准（HumanEval、MBPP、HumanEval+、MultiPL-E）和四项推理基准（GSM8K、MATH、ARC-Challenge、MMLU-STEM）上验证压缩阈值具有跨语言与难度的泛化性；第二，首次进行每令牌困惑度分析（n=723 tokens），揭示‘困惑度悖论’：代码语法令牌被保留（高困惑度），而数学问题中的数值虽关键却被修剪（低困惑度）。签名注入使通过率提升34个百分点（5.3%→39.3%；Cohen's h=0.890）；第三，提出TAAC（任务感知自适应压缩），实现22%成本降低，96%质量保留，优于固定比率压缩7%。MBPP验证（n=1,800次试验）显示系统性变化：r=0.3时为3.6%，r=1.0时达54.6%。
原文摘要 · Abstract (English)
In "Compress or Route?" (Johnson, 2026), we found that code generation tolerates aggressive prompt compression (r >= 0.6) while chain-of-thought reasoning degrades gradually. That study was limited to HumanEval (164 problems), left the "perplexity paradox" mechanism unvalidated, and provided no adaptive algorithm. This paper addresses all three gaps. First, we validate across six code benchmarks (HumanEval, MBPP, HumanEval+, MultiPL-E) and four reasoning benchmarks (GSM8K, MATH, ARC-Challenge, MMLU-STEM), confirming the compression threshold generalizes across languages and difficulties. Second, we conduct the first per-token perplexity analysis (n=723 tokens), revealing a "perplexity paradox": code syntax tokens are preserved (high perplexity) while numerical values in math problems are pruned despite being task-critical (low perplexity). Signature injection recovers +34 percentage points in pass rate (5.3% to 39.3%; Cohen's h=0.890). Third, we propose TAAC (Task-Aware Adaptive Compression), achieving 22% cost reduction with 96% quality preservation, outperforming fixed-ratio compression by 7%. MBPP validation (n=1,800 trials) confirms systematic variation: 3.6% at r=0.3 to 54.6% at r=1.0.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。