让AI学会根据成本灵活决定是否提问、猜答或枚举意图。
Learning Steerable Clarification Policies with Collaborative Self-play
- 用自对弈训练双模型，模拟用户与AI的对话决策。
- 在成本惩罚下，模型准确率提升且行为随成本可预测调整。
- 适用于语音/小屏等受限场景，支持未见过的成本值泛化。
为应对不明确或模糊的查询，AI助手需具备管理不确定性的策略，以判断何时直接猜测用户意图、何时枚举多种可能意图，或何时提出澄清问题。然而，这些策略依赖于上下文因素，如用户偏好或交互模态。例如，在小屏幕或语音场景中，枚举意图会带来操作负担。本文提出使用自对弈（self-play）训练可调节的澄清策略。通过两个代理——一个模拟用户，另一个代表AI助手——生成对话，其中用户发出潜在模糊的查询，助手需决定响应方式。关键在于，模型接收每个澄清问题的数值成本及生成词汇，目标是最大化成本惩罚后的准确率。我们采用强化自训练（ReST）方法，使模型在训练中获得高奖励，并展现出随输入成本可预测的行为变化，从而提升最终奖励与准确率。此外，该方法还能泛化至训练时未见的成本数值。
原文摘要 · Abstract (English)
To handle underspecified or ambiguous queries, AI assistants need a policy for managing their uncertainty to determine (a) when to guess the user intent and answer directly, (b) when to enumerate and answer multiple possible intents, and (c) when to ask a clarifying question. However, such policies are contextually dependent on factors such as user preferences or modality. For example, enumerating multiple possible user intentions is cumbersome on small screens or in a voice setting. In this work, we propose to train steerable policies for managing this uncertainty using self-play. Given two agents, one simulating a user and the other an AI assistant, we generate conversations where the user issues a potentially ambiguous query, and the assistant needs to determine how to respond. Importantly, the model takes as input the numerical cost of each clarification question, and each generated word, and is asked to take the action that will maximize its final reward, which is the cost-penalized accuracy. We use Reinforced Self-Training (ReST) to train our model to achieve high reward and show this leads to a steerable policy that changes its behavior predictably conditioned on the provided costs, leading to higher reward and accuracy. Moreover, our procedure also generalizes to numerical cost values that were unobserved at training time.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。