让大模型像侦探一样收集证据，避免被用户误导。
LLM-as-an-Investigator: Evidence-First Reasoning for Robust Interactive Problem Diagnosis
- 构建调查代理，先评估问题模糊性，再提问澄清
- 通过证据积累使正确解释概率超越其他选项
- 适合需要精准诊断的工程故障排查场景
大型语言模型（LLMs）越来越多地用于技术问题的交互式解决。然而，当用户给出不完整描述或看似合理但未经验证的解释时，LLMs可能过早认同这些假设并提出解决方案，而非收集充分证据。我们称这种行为为‘用户驱动的奉承’：即模型倾向于强化用户提供的假说，而非检验其他可能性。本文提出‘大模型作为调查员’（LLM-as-an-Investigator）方法，采用证据优先的智能体框架进行稳健的问题诊断。该方法由一个解决方案调查代理实现，其可估计初始问题描述的模糊性，生成候选假说，提出针对性澄清问题，并在每次回答后更新假说概率。模型不会立即响应，而是持续调查，直到某解释的证据显著优于其他选项。为评估该方法，我们从机械、电气和液压领域的已解决技术论坛帖子中构建基准数据集。采用三代理评估流程：问题-解法提取代理将已解决帖子转为结构化案例，真值评估代理模拟用户并隐藏真实解，测试助手通过对话还原解决方案。实验对比标准助手、推理导向型模型与所提调查模型在不同LLM基座上的表现。除诊断准确率外，还分析了标准助手在诊断任务中对误导性用户假设的追随现象。结果表明，所提方法比直接提示和仅推理基线更准确识别问题，其证据优先协议有效降低用户引导的对话偏差。
原文摘要 · Abstract (English)
Large language models (LLMs) are increasingly used as interactive assistants for technical problem solving. However, when users provide incomplete descriptions or plausible but unverified explanations, LLMs may prematurely align with these assumptions and propose solutions before collecting sufficient evidence. We refer to this behavior as user-driven sycophancy: the tendency of an LLM to reinforce a user-provided hypothesis instead of testing alternative explanations. This paper introduces LLM-as-an-Investigator, an evidence-first agentic AI methodology for robust problem diagnosis. The approach is implemented through a Solution Investigator Agent, which estimates the ambiguity of an initial problem description, generates candidate hypotheses, asks targeted clarification questions, and updates hypothesis probabilities after each answer. Rather than producing an immediate response, the agent continues the investigation until the evidence makes one candidate explanation stronger than the alternatives. To evaluate the approach, we build a benchmark from solved technical forum threads in mechanical, electrical, and hydraulic domains. We use a three-agent evaluation pipeline in which a Problem-Solution Extractor Agent converts solved threads into structured cases, a Ground-Truth Evaluator Agent simulates the user while hiding the known solution, and the tested assistant attempts to recover the solution through dialogue. The experiments compare standard assistants, reasoning-oriented LLMs, and the proposed investigator-based model across LLM backbones. In addition to diagnostic accuracy, we analyze how standard assistants follow misleading user hypotheses in diagnostic cases. The results show that the proposed approach identifies the problem more accurately than direct prompting and reasoning-only baselines, while its evidence-first protocol helps reduce user-induced conversational bias.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。