让大模型在推理中途放弃错误路径，节省算力并提升准确率
Knowing When to Quit: A Principled Framework for Dynamic Abstention in LLM Reasoning
- 将放弃回答建模为强化学习中的显式动作，动态判断是否提前终止推理
- 在数学推理和毒性内容识别任务中，选择性准确率优于现有方法
- 提供可调节的计算-信息权衡机制，适合对效率与准确性要求高的场景
采用思维链推理的大语言模型常因生成冗长错误回应而浪费大量计算资源。通过放弃可能不正确的输出可缓解此问题。现有放弃策略多在生成前或生成后决策，而动态中途放弃则在每个词元位置评估是否终止无望的推理路径。此前工作仅探索了经验性变体，缺乏理论指导。本文提出一种形式化分析框架，将放弃视为正则化强化学习中的显式动作，用放弃奖励参数调控计算与信息的权衡。理论证明：当价值函数低于该奖励时放弃，性能严格优于自然基线。进一步推导出高效近似价值函数的方法。在数学推理与毒性规避任务上的实验验证了理论，表明选择性准确率显著提升。
原文摘要 · Abstract (English)
LLMs utilizing chain-of-thought reasoning often waste substantial compute by producing long, incorrect responses. Abstention can mitigate this by withholding outputs unlikely to be correct. While most abstention methods decide to withhold outputs before or after generation, dynamic mid-generation abstention considers early termination of unpromising reasoning traces at each token position. Prior work has explored empirical variants of this idea, but principled guidance for the abstention rule remains lacking. We present a formal analysis of dynamic abstention for LLMs, modeling abstention as an explicit action within a regularized reinforcement learning framework. An abstention reward parameter controls the trade-off between compute and information. We show that abstaining when the value function falls below this reward strictly outperforms natural baselines under general conditions. We further derive a principled and efficient method to approximate the value function. Empirical results on mathematical reasoning and toxicity avoidance tasks support our theory and demonstrate improved selective accuracy over existing methods.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。