用强化学习生成对抗性查询，提升大模型调用外部工具的鲁棒性。
Exploring Weaknesses in Function Call Models via Reinforcement Learning: An Adversarial Data Augmentation Approach
- 通过强化学习训练查询模型，生成针对性挑战函数调用能力的对抗样本。
- 在零和博弈框架下交替训练，使函数调用模型逐步克服弱点。
- 适合研究大模型工具调用鲁棒性或安全性的研究人员参考。
函数调用能力已成为大语言模型（LLMs）与外部工具和API有效交互的关键。现有方法依赖人工标注或模型自动生成的数据对LLM进行微调，但这些方法往往缺乏针对性，受限于固定模式和数据分布，难以有效提升函数调用模型的泛化与鲁棒性。为解决此问题，我们提出一种新颖的对抗性数据增强方法，利用强化学习系统性地识别并靶向函数调用LLM的弱点。我们的训练框架引入一个通过强化学习（RL）训练的查询模型，生成专门用于挑战函数调用（FC）模型的对抗性查询。该方法采用零和博弈形式，使查询模型与FC模型进行迭代交替训练。整体上，该方法推动了更鲁棒的函数调用模型发展，并提供了一种系统性识别和纠正大模型与外部工具交互能力缺陷的方法。
原文摘要 · Abstract (English)
Function call capabilities have become crucial for Large Language Models (LLMs), enabling them to interact more effectively with external tools and APIs. Existing methods for improving the function call capabilities of LLMs rely on data obtained either through manual annotation or automated generation by models, and use this data to finetune the LLMs. However, these methods often lack targeted design and are constrained by fixed patterns and data distributions, which limits their effectiveness in enhancing the generalization and robustness of function call LLMs. To address this limitation, we propose a novel adversarial data augmentation method that employs reinforcement learning to systematically identify and target the weaknesses of function call LLMs. Our training framework introduces a query model trained with reinforcement learning (RL) to generate adversarial queries that are specifically designed to challenge function call (FC) models. This approach adopts a zero sum game formulation, where the query model and the FC model engage in iterative alternating training. Overall, our method advances the development of more robust FC models and provides a systematic way to identify and correct weaknesses in the ability of LLMs to interact with external tools.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。