通过渐进压缩揭示序列压缩的极限与语义失效机制
Progressive Cramming: Reliable Token Compression and What It Reveals
- 逐步增加压缩前缀长度，直到无法在预算内重建
- 压缩嵌入导致多项选择任务准确率下降，生成评估几乎崩溃
- 早期注意力层的交互是性能退化的关键原因
Token cramming 能以接近完美的重建精度将序列压缩为学习得到的嵌入，但固定的令牌预算和99%的准确率阈值使得残余误差究竟是优化失败还是根本限制尚不明确。我们提出渐进压缩方法，逐个增加目标前缀令牌，仅在无法在固定优化预算内实现重建时停止。渐进轨迹在嵌入空间中呈现出低维结构。在原始前缀仍可用的情况下，附加压缩嵌入会导致多项选择基准上出现适度但一致的准确率下降，而在生成评估中则几乎完全丧失能力。因果注意力剔除干预表明，这种退化源于嵌入在模型早期层中的交互。这些结果将渐进压缩定位为研究压缩极限的工具，并表明完美重建——可通过脆弱的引导实现而非可迁移的语义——不足以构成有意义的压缩。
原文摘要 · Abstract (English)
Token cramming compresses sequences into learned embeddings with near-perfect reconstruction, but fixed token budgets and 99\% accuracy thresholds leave it unclear whether residual errors reflect optimization failures or fundamental limits. We introduce progressive cramming, which grows the target prefix token-by-token, stopping only when reconstruction is no longer achievable within a fixed optimization budget. Progressive trajectories occupy low-dimensional structure in embedding space. Prepending a crammed embedding causes a moderate but consistent accuracy drop on multiple-choice benchmarks even with the original prefix in context, and collapses capability almost entirely under generative evaluation. Causal attention-knockout interventions trace this degradation to the embedding's interactions in the model's early layers. These results position progressive cramming as a tool for studying compression limits and show that perfect reconstruction - achievable through brittle steering rather than transferable semantics - is insufficient for meaningful compression.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。