用可回放的多智能体系统,探测大模型在医疗诊断中的信念形成过程。
Ask WhAI:Probing Belief Formation in Role-Primed LLM Agents
- 构建可记录、重放和干预智能体信念的系统框架
- 发现模型受角色先验影响,易固守既定观点并排斥反证据
- 适合研究人工智能推理偏差与跨领域认知壁垒
我们提出 Ask WhAI,一个用于检查和操纵多智能体交互中信念状态的系统级框架。该框架可记录并重放智能体交互,支持对每个智能体信念与推理过程的独立查询,并能注入反事实证据以测试信念结构对新信息的响应。我们将该框架应用于一个具有多智能体共享记忆(时间戳电子病历,EMR)和一个仅在被主动查询时才揭示真实结果的实验室代理(LabAgent)的医学案例模拟器。在一名儿童突发神经精神症状的多专科诊断过程中,大语言模型智能体各自被强角色先验(如“像神经科医生一样行动”)引导,通过共享病历与协调员进行顺序或并行交流。关键诊断节点设置断点,实现事件前后的信念查询,从而区分根深蒂固的角色偏见与真实推理或证据整合效应。模拟结果显示,智能体信念常反映现实学科立场,包括过度依赖经典研究、对反证据的抵抗等,且这些信念可被追踪与质询,这是人类专家难以做到的。通过使此类动态可见且可测试,Ask WhAI 提供了一种可复现的研究多智能体科学推理中信念形成与知识孤岛的方法。
原文摘要 · Abstract (English)
We present Ask WhAI, a systems-level framework for inspecting and perturbing belief states in multi-agent interactions. The framework records and replays agent interactions, supports out-of-band queries into each agent's beliefs and rationale, and enables counterfactual evidence injection to test how belief structures respond to new information. We apply the framework to a medical case simulator notable for its multi-agent shared memory (a time-stamped electronic medical record, or EMR) and an oracle agent (the LabAgent) that holds ground truth lab results revealed only when explicitly queried. We stress-test the system on a multi-specialty diagnostic journey for a child with an abrupt-onset neuropsychiatric presentation. Large language model agents, each primed with strong role-specific priors ("act like a neurologist", "act like an infectious disease specialist"), write to a shared medical record and interact with a moderator across sequential or parallel encounters. Breakpoints at key diagnostic moments enable pre- and post-event belief queries, allowing us to distinguish entrenched priors from reasoning or evidence-integration effects. The simulation reveals that agent beliefs often mirror real-world disciplinary stances, including overreliance on canonical studies and resistance to counterevidence, and that these beliefs can be traced and interrogated in ways not possible with human experts. By making such dynamics visible and testable, Ask WhAI offers a reproducible way to study belief formation and epistemic silos in multi-agent scientific reasoning.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。