让大模型学会该思考时深思，该省时就直接回答。
HiPO: Hybrid Policy Optimization for Dynamic Reasoning in LLMs
- 用混合数据+强化学习，让模型自适应选择深度推理或快速作答。
- 数学与编程任务中，令牌消耗减少超30%，准确率不降反升。
- 适合资源有限场景，提升大模型推理效率的实用方案。
大型语言模型（LLMs）越来越多地依赖思维链（Chain-of-Thought, CoT）推理来提升复杂任务的准确性。然而，始终生成冗长的推理过程效率低下，导致令牌使用过多且推理成本升高。本文提出混合策略优化（HiPO），一种自适应推理控制框架，使LLM能选择性地决定何时进行详细推理（Think-on），何时直接作答（Think-off）。具体而言，HiPO结合了提供成对Think-on与Think-off响应的混合数据管道，以及平衡准确率与效率的混合强化学习奖励机制，避免过度依赖详细推理。在数学与编程基准测试中的实验表明，HiPO可显著减少令牌长度，同时保持或提升准确率。我们希望HiPO能成为高效自适应推理的原理性方法，推动面向推理的LLM在真实、资源敏感环境中的部署。
原文摘要 · Abstract (English)
Large Language Models (LLMs) increasingly rely on Chain-of-Thought (CoT) reasoning to improve accuracy on complex tasks. However, always generating lengthy reasoning traces is inefficient, leading to excessive token usage and higher inference costs. This paper introduces the Hybrid Policy Optimization (i.e., HiPO), a framework for adaptive reasoning control that enables LLMs to selectively decide when to engage in detailed reasoning (Think-on) and when to respond directly (Think-off). Specifically, HiPO combines a hybrid data pipelineproviding paired Think-on and Think-off responseswith a hybrid reinforcement learning reward system that balances accuracy and efficiency while avoiding over-reliance on detailed reasoning. Experiments across mathematics and coding benchmarks demonstrate that HiPO can substantially reduce token length while maintaining or improving accuracy. Finally, we hope HiPO a can be a principled approach for efficient adaptive reasoning, advancing the deployment of reasoning-oriented LLMs in real-world, resource-sensitive settings.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。