用熵抑制技术压缩大模型流水线中的冗余标记，显著节省计算资源。
Entropy Gate: Entropy Quenching for Near-Lossless Token Compression in LLM Pipelines
- 基于多维度信息能量评估每个标记，按熵值逐步剔除低效内容。
- 在保持语义相似度高于0.80的前提下，实现40%-60%的压缩率。
- 无需修改模型即可部署，适合需要高效推理的智能体场景。
大模型流水线中大量标记被浪费在低信息内容上：重复上下文、冗长回复和重复模板。本文提出熵门（Entropy Gate）框架，采用熵抑制机制——一种渐进冻结低能量标记而保留语义完整性的热力学过程。每个标记获得包含统计、结构与位置因素的综合信息能量 $E(t)$。通过自适应降温曲线 $T(τ) = T_0 / (1 + ατ)$，移除玻尔兹曼存活概率 $p_i = \ ext{exp}(-E_i / kT)$ 低于阈值的标记，并设置保真度门限，当能量加权相似度降至 $θ$ 以下时停止压缩。理论证明按 $E(t)$ 降序选择标记可最大化预期语义保留，且压缩产生嵌套生存集合，可达信息论极限 $ ext{CR} \to 1 - I(P; T)/H(P)$。初步启发式方法在五类提示中实现40%-60%压缩率，语义保真度 $S_E > 0.80$；引入能量平方放大 $E \to E^2$ 可额外提升10%-25个百分点。上下文去重对重复块再节省50%-70%。输出端压缩受简洁性提升准确性的启发，进一步降低响应开销。结合外部记忆后，压缩率在智能体任务中达到88%-96%，且为无状态、模型无关设计，可作为OpenAI兼容的HTTP代理部署。
原文摘要 · Abstract (English)
LLM pipelines waste substantial token budgets on low-information content: repeated context, verbose responses, and redundant boilerplate. We introduce Entropy Gate, a token compression framework applying entropy quenching $-$ a thermodynamic process that progressively freezes out low-energy tokens while preserving semantic fidelity. Each token receives a multi-factor information energy $E(t)$ combining statistical, structural, and positional components. An adaptive quenching schedule $T(τ) = T_0 / (1 + ατ)$ removes tokens whose Boltzmann survival probability $p_i = \exp(-E_i / kT)$ falls below threshold, with a fidelity gate halting compression when energy-weighted similarity drops below $θ$. We prove token selection by descending $E(t)$ maximizes expected semantic preservation, that quenching produces nested survival sets, and that achievable compression approaches the information-theoretic limit $\text{CR} \to 1 - I(P; T)/H(P)$. A Phase 1 heuristic achieves 40-60% compression across five prompt categories while maintaining $S_E > 0.80$, with energy-squared amplification $E \to E^2$ adding 10-25 percentage points. Context deduplication adds 50-70% savings on repeated blocks. Output-side quenching, motivated by findings that brevity improves accuracy, further reduces response overhead. Combined with external memory, reduction composes multiplicatively to 88-96% for agentic workloads. The framework is stateless, model-agnostic, and deploys as an OpenAI-compatible HTTP proxy.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。