arXiv:2507.15758cs.AIcs.CL2025-07被引 12

让大模型自动判断推理长度,又快又准

LAPO: Internalizing Reasoning Efficiency via Length-Adaptive Policy Optimization

  • 用两阶段强化学习让模型学会自适应推理深度
  • 数学推理任务中减少40.9%的令牌消耗,准确率提升2.3%
  • 适合追求高效推理的AI系统开发者

大型推理模型通过扩展思维链序列取得了显著性能,但这种计算自由导致简单问题也生成过多令牌。我们提出长度自适应策略优化(LAPO),将推理长度控制从外部约束转化为模型内在能力。与现有方法依赖刚性限制或事后干预不同,LAPO通过两阶段强化学习,使模型学会发现成功解题长度的统计分布,并将其作为元认知引导嵌入推理上下文,实现推理时的灵活性。在数学推理基准测试中,LAPO将令牌使用量最多减少40.9%,同时准确率提升2.3%。分析表明,经LAPO训练的模型能根据问题复杂度动态分配计算资源,实现高效推理而不牺牲质量。

原文摘要 · Abstract (English)

Large reasoning models have achieved remarkable performance through extended chain-of-thought sequences, yet this computational freedom leads to excessive token generation even for simple problems. We present Length-Adaptive Policy Optimization (LAPO), a novel framework that transforms reasoning length control from an external constraint into an intrinsic model capability. Unlike existing approaches that impose rigid limits or rely on post-hoc interventions, LAPO enables models to internalize an understanding of appropriate reasoning depth through a two-stage reinforcement learning process. In the first stage, models learn natural reasoning patterns by discovering the statistical distribution of successful solution lengths. The second stage leverages these patterns as meta-cognitive guidance, embedding them directly within the model's reasoning context to ensure inference-time flexibility. Experiments on mathematical reasoning benchmarks demonstrate that LAPO reduces token usage by up to 40.9% while improving accuracy by 2.3%. Our analysis reveals that models trained with LAPO develop emergent abilities to allocate computational resources based on problem complexity, achieving efficient reasoning without sacrificing quality.

推理优化强化学习效率提升

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。