用对话优化提升工具增强大模型的交互能力，无需人工标注。
DiaTool-DPO: Multi-Turn Direct Preference Optimization for Tool-Augmented Large Language Models
- 将对话建模为5种状态的马尔可夫决策过程，按轨迹分类用户请求。
- 自动构建正确与错误对话流的配对数据集，实现94.8%信息获取率。
- 无需专家示范即可训练，适合真实场景下复杂任务的工具调用。
工具增强型大语言模型（TA-LLMs）在实际应用中展现出潜力，但面临处理不完整查询和超出范围请求的挑战。现有方法主要依赖专家轨迹的监督微调，本文提出DiaTool-DPO，一种通过直接偏好优化增强TA-LLM对话能力的新方法。我们将TA-LLM交互建模为具有5种不同对话状态的马尔可夫决策过程，并根据状态转移轨迹将用户请求分为3类。我们自动构建正确与错误对话流程的配对轨迹数据集，并引入专用目标损失函数以实现对话控制。全面评估表明，DiaTool-DPO在信息获取方面达到94.8%，工具调用拒绝率达91%，相比基线分别提升44%和9.6%，接近GPT-4o性能，同时保持核心功能。该方法为开发无需额外专家演示或人工标注即可应对多样现实场景的TA-LLMs开辟了新路径。
原文摘要 · Abstract (English)
Tool-Augmented Larage Language Models (TA-LLMs) have shown promise in real-world applications, but face challenges in handling incomplete queries and out-of-scope requests. While existing approaches rely mainly on Supervised Fine-Tuning with expert trajectories, we propose DiaTool-DPO, a novel method that enhances TA-LLM's dialogue capabilities through Direct Preference Optimization. We model TA-LLM interactions as a Markov Decision Process with 5 distinct dialogue states and categorize user queries into 3 types based on their state transition trajectories. We automatically construct paired trajectory datasets of correct and incorrect dialogue flows and introduce a specialized objective loss for dialogue control. Our comprehensive evaluation demonstrates that DiaTool-DPO approaches GPT-4o's performance (94.8% in information gathering, 91% in tool call rejection) with substantial improvements over baseline (44% and 9.6% respectively) while maintaining core functionality. Our approach opens new possibilities for developing TA-LLMs that can handle diverse real-world scenarios without requiring additional expert demonstrations or human labeling.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。