让大模型主动提问,解决任务描述不清的问题
Active Task Disambiguation with LLMs
- 通过贝叶斯实验设计,让模型主动生成最有效的澄清问题
- 相比只在问题空间内推理的方法,能更高效缩小可行解空间
- 适合需要精准理解用户意图的交互式AI系统
尽管大型语言模型在各类基准测试中表现优异,其处理现实交互中常见模糊任务描述的能力仍待深入探索。为此,我们提出了任务模糊性的形式化定义,并将任务消歧问题置于贝叶斯实验设计的框架下。通过提出澄清性问题,语言模型代理能够获取额外的任务规范,逐步缩小可行解空间,降低生成不理想输出的风险。然而,生成有效澄清问题需要模型具备元认知推理能力,这可能是当前大模型尚不具备的。我们提出的主动任务消歧方法使模型能够生成最大化信息增益的目标问题,实质上将推理负担从隐式转向显式。实证结果表明,这种问题选择方式在任务消歧效果上优于仅依赖问题空间内推理的方法。
原文摘要 · Abstract (English)
Despite the impressive performance of large language models (LLMs) across various benchmarks, their ability to address ambiguously specified problems--frequent in real-world interactions--remains underexplored. To address this gap, we introduce a formal definition of task ambiguity and frame the problem of task disambiguation through the lens of Bayesian Experimental Design. By posing clarifying questions, LLM agents can acquire additional task specifications, progressively narrowing the space of viable solutions and reducing the risk of generating unsatisfactory outputs. Yet, generating effective clarifying questions requires LLM agents to engage in a form of meta-cognitive reasoning, an ability LLMs may presently lack. Our proposed approach of active task disambiguation enables LLM agents to generate targeted questions maximizing the information gain. Effectively, this approach shifts the load from implicit to explicit reasoning about the space of viable solutions. Empirical results demonstrate that this form of question selection leads to more effective task disambiguation in comparison to approaches relying on reasoning solely within the space of questions.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。