让大模型学会精简推理,简单问题不啰嗦,复杂问题不失深度。
DRQA: Dynamic Reasoning Quota Allocation for Controlling Overthinking in Reasoning Large Language Models
- 通过强化学习训练模型动态分配推理资源,根据问题难度自动调整思考长度。
- 在多个数学与科学推理基准上,令牌消耗减少30%以上,准确率反而提升或持平。
- 适合需要高效部署大模型的场景,如实时问答、边缘计算等应用者参考。
推理型大语言模型(RLLMs)如OpenAI-O3和DeepSeek-R1虽具备结构化多步推理能力,但常因过度思考而生成冗长推理链,导致令牌消耗过多、计算效率低下。我们观察到,在批量处理时,模型会因隐式资源竞争而自动压缩简单问题的推理步骤。受此启发,提出动态推理配额分配(DRQA)方法,将批处理中的资源竞争机制迁移至单个问题推理中。DRQA利用批量生成的偏好数据与强化学习,训练模型内化对准确且简洁回答的偏好,使模型能针对简单问题生成简短答案,同时为复杂问题保留充分推理深度。在广泛数学与科学推理基准上的实验表明,DRQA显著降低令牌使用量,同时维持甚至提升准确率。该方法有效缓解了过度思考问题,为更高效、可扩展的RLLM部署提供了新方向。
原文摘要 · Abstract (English)
Reasoning large language models (RLLMs), such as OpenAI-O3 and DeepSeek-R1, have recently demonstrated remarkable capabilities by performing structured and multi-step reasoning. However, recent studies reveal that RLLMs often suffer from overthinking, i.e., producing unnecessarily lengthy reasoning chains even for simple questions, leading to excessive token consumption and computational inefficiency. Interestingly, we observe that when processing multiple questions in batch mode, RLLMs exhibit more resource-efficient behavior by dynamically compressing reasoning steps for easier problems, due to implicit resource competition. Inspired by this, we propose Dynamic Reasoning Quota Allocation (DRQA), a novel method that transfers the benefits of resource competition from batch processing to single-question inference. Specifically, DRQA leverages batch-generated preference data and reinforcement learning to train the model to allocate reasoning resources adaptively. By encouraging the model to internalize a preference for responses that are both accurate and concise, DRQA enables it to generate concise answers for simple questions while retaining sufficient reasoning depth for more challenging ones. Extensive experiments on a wide range of mathematical and scientific reasoning benchmarks demonstrate that DRQA significantly reduces token usage while maintaining, and in many cases improving, answer accuracy. By effectively mitigating the overthinking problem, DRQA offers a promising direction for more efficient and scalable deployment of RLLMs, and we hope it inspires further exploration into fine-grained control of reasoning behaviors.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。