提出首个工具遗忘方法，让大模型安全删除工具知识。
Tool Unlearning for Tool-Augmented LLMs
- 设计ToolDelete框架，通过参数优化实现工具知识移除。
- 在多个数据集上验证，可精准删除指定工具且不影响其他能力。
- 引入新攻击评估方法，确保遗忘效果可量化验证。
工具增强型大语言模型通常在查询-响应对数据集上训练，将使用工具或API的能力嵌入模型参数知识中。由于存在安全漏洞、隐私法规或工具淘汰等问题，模型需具备遗忘已学习工具的能力。然而，现有去学习研究尚未涉及‘工具遗忘’这一新任务。本文首次提出该任务，并指出其与传统去学习的不同：需移除知识而非单个样本，优化成本高，且缺乏合理评估指标。为此，我们提出ToolDelete，首个针对工具增强型大模型的去学习方法，具备三大核心特性以应对挑战，并引入新的成员推断攻击（MIA）模型用于有效评估。在多个工具学习数据集和工具增强型大模型上的实验证明，ToolDelete能有效遗忘随机选取的工具，同时保留非删除工具的知识及通用任务性能。
原文摘要 · Abstract (English)
Tool-augmented large language models (LLMs) are often trained on datasets of query-response pairs, which embed the ability to use tools or APIs directly into the parametric knowledge of LLMs. Tool-augmented LLMs need the ability to forget learned tools due to security vulnerabilities, privacy regulations, or tool deprecations. However, ``tool unlearning'' has not been investigated in unlearning literature. We introduce this novel task, which requires addressing distinct challenges compared to traditional unlearning: knowledge removal rather than forgetting individual samples, the high cost of optimizing LLMs, and the need for principled evaluation metrics. To bridge these gaps, we propose ToolDelete, the first approach for unlearning tools from tool-augmented LLMs. It implements three key properties to address the above challenges for effective tool unlearning and introduces a new membership inference attack (MIA) model for effective evaluation. Extensive experiments on multiple tool learning datasets and tool-augmented LLMs show that ToolDelete effectively unlearns randomly selected tools, while preserving the LLM's knowledge on non-deleted tools and maintaining performance on general tasks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。