用多模态游戏检测大模型聊天是否真实反映所见所行
QUACK: Questioning, Understanding, and Auditing Communicated Knowledge in Multimodal Social Deduction Agents

- 通过游戏日志重建真实行为轨迹,逐句验证发言真实性
- 强模型仍有15.1%空间描述错误,11.5%指控无依据
- 适合研究模型可信度、对抗性推理与多模态对齐的学者
社交推理游戏已成为检验大语言模型代理在推理、欺骗、协作和信念建模方面能力的重要测试平台。然而,多数环境仅以胜负率等游戏结果评分,且主要依赖文本交互,难以判断代理的语言是否真正基于其感知与行动,也无法识别其行为失效模式。为此,我们提出QUACK——一个开源环境与评估框架,用于审计多模态社交推理中代理语言的可证伪性。QUACK从三个层面评估代理:游戏结果、行为轨迹和话语一致性。其核心陈述验证流水线从引擎日志重构每个代理的真实行为轨迹,并将每条讨论内容与其比对,自动标记空间幻觉、无依据指控、欺骗崩溃及语言-动作不一致等问题。在同质与跨模型对抗设置下评估三种前沿视觉语言模型,发现即使最强模型也有15.1%可验证的空间陈述存在幻觉,11.5%的指控完全缺乏支持。相关完整引擎、评估框架、工具包与日志已公开于https://github.com/AAAAA-Academia-Attractions/QUACK。
原文摘要 · Abstract (English)
Social deduction games have become a popular testbed for probing reasoning, deception, coordination, and belief modeling in Large Language Model (LLM) agents. However, most environments are scored only by game outcomes such as win rates and largely remain to text-only interaction, making it difficult to tell whether an agent's language is actually grounded in what it perceived and did, or to identify the failure modes underlying its behavior. To address this gap, we introduce QUACK, an open-source environment and evaluation framework for auditing the grounding of agent language in multimodal social reasoning. QUACK evaluates agents at three levels: game outcomes, behavioral trajectories, and utterance-level consistency. Its core Statement Verification Pipeline reconstructs each agent's ground-truth trajectory from engine logs and checks every discussion claim against it, automatically flagging spatial hallucination, unsupported accusation, deception collapse, and language-action inconsistency. Evaluating three frontier VLMs in both homogeneous and cross-model adversarial settings, we find that even the strongest agent hallucinates 15.1% of its verifiable spatial claims and 11.5% of accusations are strictly unsupported. We release the full engine, evaluation framework, toolkit, and logs in https://github.com/AAAAA-Academia-Attractions/QUACK.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。