arXiv:2511.04393cs.AI2025-11被引 5

通过反复优化低损失决策路径,让大模型更会做决策。

Post-Training LLMs as Better Decision-Making Agents: A Regret-Minimization Approach

  • 用后悔值引导模型自我迭代优化决策轨迹。
  • 在多种任务中显著降低决策后悔值,提升探索与利用平衡。
  • 无需人工模板,适合各类复杂场景和大模型部署。

大语言模型(LLM)被越来越多用于交互式动态环境中的决策任务,但因其并非专为决策设计,现有研究显示其在基础在线决策问题中仍表现不佳,难以实现低后悔值或有效探索-利用权衡。为此,我们提出迭代后悔最小化微调(Iterative RMFT),一种后训练方法:模型多次生成决策轨迹,选择后悔值最低的k条,再以此微调自身。相比以往依赖已知算法或人工思维链模板的方法,本方法利用后悔值作为信号,激发模型自主决策能力和推理过程,避免僵硬输出设计,提供更自然、灵活的训练信号。实验证明,Iterative RMFT在多种模型上均显著提升决策性能,涵盖具有数值输入输出的Transformer、开源权重模型及GPT-4o mini等闭源先进模型。其对输出与推理格式的高度灵活性,使模型可泛化至不同时间跨度、动作空间、奖励机制和自然语言上下文的任务。最后,理论分析表明,在简化设定下,单层Transformer经此框架可成为无后悔学习者。总体而言,Iterative RMFT提供了一个原则性强、通用性高的后训练框架,用于增强大模型的决策能力。

原文摘要 · Abstract (English)

Large language models (LLMs) are increasingly deployed as "agents" for decision-making (DM) in interactive and dynamic environments. Yet, since they were not originally designed for DM, recent studies show that LLMs can struggle even in basic online DM problems, failing to achieve low regret or an effective exploration-exploitation tradeoff. To address this, we introduce Iterative Regret-Minimization Fine-Tuning (Iterative RMFT), a post-training procedure that repeatedly distills low-regret decision trajectories back into the base model. At each iteration, the model rolls out multiple decision trajectories, selects the k-lowest regret ones, and fine-tunes itself on them. Unlike prior methods that (a) distill action sequences from known DM algorithms or (b) rely on manually crafted chain-of-thought templates, our approach leverages the regret metric to elicit the model's own DM ability and reasoning rationales. This reliance on model-generated reasoning avoids rigid output engineering and provides more flexible, natural-language training signals. Empirical results show that Iterative RMFT improves LLMs' DM performance across diverse models - from Transformers with numerical input/output, to open-weight LLMs, and advanced closed-weight models like GPT-4o mini. Its flexibility in output and reasoning formats enables generalization across tasks with varying horizons, action spaces, reward processes, and natural-language contexts. Finally, we provide theoretical insight showing that a single-layer Transformer under this paradigm can act as a no-regret learner in a simplified setting. Overall, Iterative RMFT offers a principled and general post-training framework for enhancing LLMs' decision-making capabilities.

决策优化大模型后悔最小化后训练

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。