arXiv:2604.00304cs.CLcs.AI2026-04被引 4

用小模型监督大模型对话,提升多轮交互可靠性。

Asymmetric Actor-Critic for Multi-turn LLM Agents

论文配图:Asymmetric Actor-Critic for Multi-turn LLM Agents
图 1 · 摘自论文原文
  • 大模型作对话主体,小模型实时监控并干预。
  • 在τ-bench和UserBench上任务成功率显著提升。
  • 适合无法重试的场景,且可适配闭源大模型。

大型语言模型具备强大的推理与对话能力,但在多轮交互中保证可靠行为仍具挑战。在许多实际应用中,系统必须在单次尝试中成功,无法重试。现有方法或依赖反思或事后评估,需额外尝试;或假设模型完全可训练,无法利用专有大模型。本文提出一种非对称的演员-评论家框架,让强大的专有大模型作为演员,小型开源评论家提供运行时监督,在同一对话轨迹内监控并干预演员行为。不同于基于训练的演员-评论家方法,本框架监督一个固定的演员,在开放对话环境中运行。设计利用生成与验证的不对称性:高质量生成需大模型,而有效监督可用小模型完成。我们还提出一种数据生成流程,无需修改演员即可生成评论家微调的监督信号。在τ-bench和UserBench上的实验表明,该方法显著提升可靠性与任务成功率。此外,轻量级开源评论家表现媲美甚至超越更大专有模型,且评论家微调带来进一步增益,优于多个前沿方法。

原文摘要 · Abstract (English)

Large language models (LLMs) exhibit strong reasoning and conversational abilities, but ensuring reliable behavior in multi-turn interactions remains challenging. In many real-world applications, agents must succeed in one-shot settings where retries are impossible. Existing approaches either rely on reflection or post-hoc evaluation, which require additional attempts, or assume fully trainable models that cannot leverage proprietary LLMs. We propose an asymmetric actor-critic framework for reliable conversational agents. A powerful proprietary LLM acts as the actor, while a smaller open-source critic provides runtime supervision, monitoring the actor's actions and intervening within the same interaction trajectory. Unlike training-based actor-critic methods, our framework supervises a fixed actor operating in open-ended conversational environments. The design leverages a generation-verification asymmetry: while high-quality generation requires large models, effective oversight can often be achieved by smaller ones. We further introduce a data generation pipeline that produces supervision signals for critic fine-tuning without modifying the actor. Experiments on $τ$-bench and UserBench show that our approach significantly improves reliability and task success over strong single-agent baselines. Moreover, lightweight open-source critics rival or surpass larger proprietary models in the critic role, and critic fine-tuning yields additional gains over several state-of-the-art methods.

LLM代理对话系统监督机制非对称架构

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。