无需人工反馈，让对话模型自己优化自己。
Sparse Rewards Can Self-Train Dialogue Agents
- 用模拟环境生成稀疏奖励，让模型基于自身输出自我对齐
- 在ToolWOZ上训练后，小模型和前沿模型都显著提升工具调用能力
- 适合研究自进化AI或低成本训练对话系统的团队
当前先进大语言模型（LLM）代理在多轮对话任务中的进步主要依赖监督微调和高质量人工反馈。然而，随着基础模型能力提升，获取有意义的人工反馈变得愈发困难且成本高昂。在某些领域，基础模型甚至可能超越人类能力，使传统反馈驱动方法不再适用。本文提出一种新型自提升范式，使LLM代理可在无外部人工反馈的情况下自主优化性能。我们提出的JOSH（Juxtaposed Outcomes for Simulation Harvesting）是一种自对齐算法，利用稀疏奖励模拟环境提取理想行为，并用于训练模型自身输出。我们构建了基于MultiWOZ的稀疏奖励工具调用模拟环境ToolWOZ。实验表明，使用JOSH训练的模型，无论大小，在工具调用交互中均有显著提升，同时保持在多个基准测试上的通用能力。代码与数据已公开于GitHub。
原文摘要 · Abstract (English)
Recent advancements in state-of-the-art (SOTA) Large Language Model (LLM) agents, especially in multi-turn dialogue tasks, have been primarily driven by supervised fine-tuning and high-quality human feedback. However, as base LLM models continue to improve, acquiring meaningful human feedback has become increasingly challenging and costly. In certain domains, base LLM agents may eventually exceed human capabilities, making traditional feedback-driven methods impractical. In this paper, we introduce a novel self-improvement paradigm that empowers LLM agents to autonomously enhance their performance without external human feedback. Our method, Juxtaposed Outcomes for Simulation Harvesting (JOSH), is a self-alignment algorithm that leverages a sparse reward simulation environment to extract ideal behaviors and further train the LLM on its own outputs. We present ToolWOZ, a sparse reward tool-calling simulation environment derived from MultiWOZ. We demonstrate that models trained with JOSH, both small and frontier, significantly improve tool-based interactions while preserving general model capabilities across diverse benchmarks. Our code and data are publicly available on GitHub at https://github.com/asappresearch/josh-llm-simulation-training
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。