用强化学习压缩大模型推理过程，省时省力还保准。
HMPO: Hybrid Median-length Policy Optimization for Chain-of-Thought Compression
- 单阶段强化学习，自动调节推理长度，无需人工设限。
- 在9B到122B模型上实现19%–46%的推理词元压缩，准确率几乎不变。
- 适合需要高效推理的大模型应用，尤其数学、编程任务。
大语言模型通过长链式思维（CoT）推理取得优异表现，但带来巨大推理开销。现有压缩方法存在手动设定长度预算僵化、多阶段训练成本高、仅适用于小模型等问题。我们提出HMPO（混合中位长度策略优化），一种低成本、单阶段强化学习框架。HMPO通过三个协同组件高效压缩CoT：基于成功回放的自适应中位数预算，消除人工调参；余弦衰减词奖励，实现平滑长度惩罚；乘法奖励机制，显著抑制虚假奖励劫持，严格保障答案正确性。仅在数学数据上训练，即可跨数学、代码、科学和指令遵循任务无缝泛化。在9B至122B参数的密集型与混合专家（MoE）架构上大规模实验表明，HMPO实现19%–46%的词元压缩，准确率损失极小，且训练成本远低于现有多阶段基线。
原文摘要 · Abstract (English)
Large language models achieve remarkable performance via extended chain-of-thought (CoT) reasoning, yet this lengthy process incurs substantial inference overhead. Existing CoT compression methods struggle with inflexible manual length budgets, computationally expensive multi-stage training pipelines, and fragile scalability restricted to small models. We propose HMPO (Hybrid Median-length Policy Optimization), a cost-effective, single-stage reinforcement learning framework. HMPO efficiently compresses CoT via three synergistic components: an adaptive median-based budget derived from successful rollouts to eliminate manual tuning, a cosine-decay token reward for smooth length penalization, and a multiplicative reward formulation that substantially mitigates trivial reward hacking by strictly prioritizing answer correctness. Trained exclusively on mathematical data, HMPO generalizes seamlessly across math, code, science, and instruction-following tasks. Extensive experiments scaling from 9B to 122B parameters across dense and Mixture-of-Experts (MoE) architectures demonstrate that HMPO achieves 19%--46% token compression with negligible accuracy degradation, all while drastically reducing training costs compared to existing multi-stage baselines.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。