用任务特征优化大模型工具使用,仅需1217条数据即达顶尖性能
TL-Training: A Task-Feature-Based Framework for Training Large Language Models in Tool Use
- 基于任务特征动态加权关键token,修正训练数据缺陷
- 仅用1217条数据,工具使用表现超越多数开源与闭源模型
- 针对常见错误设计奖励机制,适合高鲁棒性工具调用场景
大型语言模型通过调用工具与环境交互,在迈向通用人工智能中取得显著进展。然而,传统监督微调方法依赖大规模数据集,常忽略工具使用中的任务特性,导致性能瓶颈。我们分析三种现有LLM,发现训练数据可能抑制工具使用行为,令牌重要性分布不均,且工具调用错误集中于少数类别。基于此,提出TL-Training框架:通过任务特征缓解劣质训练数据影响,动态调整令牌权重以聚焦关键信息,并设计针对错误类别的鲁棒奖励机制,结合近端策略优化进行优化。在CodeLLaMA-2-7B上验证,仅使用1,217条训练数据,模型在四个开源测试集上的工具使用表现匹配或超越多数开闭源模型,同时提升噪声环境下的鲁棒性与泛化能力,为大模型工具使用训练提供高效可扩展方案。代码与数据见https://github.com/Junjie-Ye/TL-Training。
原文摘要 · Abstract (English)
Large language models (LLMs) achieve remarkable advancements by leveraging tools to interact with environments, a critical step toward generalized AI. However, the standard supervised fine-tuning (SFT) approach, which relies on large-scale datasets, often overlooks task-specific characteristics in tool use, leading to performance bottlenecks. To address this issue, we analyze three existing LLMs and uncover key insights: training data can inadvertently impede tool-use behavior, token importance is distributed unevenly, and errors in tool calls fall into a small set of categories. Building on these findings, we propose~\emph{TL-Training}, a task-feature-based framework that mitigates the effects of suboptimal training data, dynamically adjusts token weights to prioritize key tokens during SFT, and incorporates a robust reward mechanism tailored to error categories, optimized through proximal policy optimization. We validate TL-Training by training CodeLLaMA-2-7B and evaluating it on four open-source test sets. Our results demonstrate that the LLM trained by our method matches or surpasses both open- and closed-source LLMs in tool-use performance using only 1,217 training data points. Additionally, our method enhances robustness in noisy environments and improves general task performance, offering a scalable and efficient paradigm for tool-use training in LLMs. Code and data are available at https://github.com/Junjie-Ye/TL-Training.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。