用强化学习提升大模型函数调用能力，让推理更稳定高效
Reasoning through Exploration: A Reinforcement Learning Framework for Robust Function Calling
- 通过熵增强优势函数，鼓励多样化的推理路径
- 在BFCL榜单上40亿参数模型超越GPT-4o和Gemini-2.5
- 适合需要可靠工具调用的复杂任务应用
大语言模型在函数调用任务中训练面临核心挑战：如何在探索复杂推理路径与稳定策略优化之间取得平衡。传统监督微调（SFT）难以培养稳健推理能力，而常规强化学习（RL）则存在探索效率低的问题。本文提出基于组相对策略优化（GRPO）的新型框架EGPO，其核心是将模型思维链（CoT）的熵纳入策略梯度计算的优势函数中，以促进多样化推理策略生成。为保持优化方向，熵奖励通过裁剪机制进行约束。结合严格的二元奖励信号，EGPO有效引导模型发现结构化且准确的工具调用模式。在具有挑战性的伯克利函数调用排行榜（BFCL）上，使用EGPO训练的40亿参数模型达到同类规模新纪录，超越包括GPT-4o和Gemini-2.5在内的多个强基线。
原文摘要 · Abstract (English)
The effective training of Large Language Models (LLMs) for function calling faces a critical challenge: balancing exploration of complex reasoning paths with stable policy optimization. Standard methods like Supervised Fine-Tuning (SFT) fail to instill robust reasoning, and traditional Reinforcement Learning (RL) struggles with inefficient exploration. We propose \textbf{EGPO}, a new RL framework built upon Group Relative Policy Optimization (GRPO), designed to address this challenge directly. The core of EGPO is an entropy-enhanced advantage function that integrates the entropy of the model's Chain-of-Thought (CoT) into the policy gradient computation. This encourages the generation of diverse reasoning strategies. To maintain optimization direction, the entropy bonus is carefully constrained by a clipping mechanism. Complemented by a strict, binary reward signal, EGPO effectively guides the model towards discovering structured and accurate tool invocation patterns. On the challenging Berkeley Function Calling Leaderboard (BFCL), a 4B-parameter model trained with EGPO sets a new state-of-the-art among models of comparable size, surpassing a range of strong competitors, including GPT-4o and Gemini-2.5.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。