让多模态大模型推理更忠实于视觉证据，减少幻觉。
Faithful-First Reasoning, Planning, and Acting for Multimodal LLMs
- 先评估中间推理是否忠实于视觉信息，再据此规划行动
- 在多个基准上使感知忠实度提升最高24%，且不降低准确率
- 适合关注多模态推理可信度的研究者与开发者
多模态大语言模型常因推理偏离视觉证据或与最终预测矛盾而出现不忠实问题。本文提出‘忠实优先’推理、规划与执行（Faithful-First RPA）框架，其中 FaithEvi 通过逐步和链级监督评估中间推理的忠实性，FaithAct 利用这些信号在推理过程中规划并执行忠于事实的动作。在多个多模态推理基准上的实验表明，该框架相比基于提示和工具增强的推理方法，感知忠实度最高提升24%，同时保持任务准确率。分析显示，将忠实性作为核心指导原则，可生成更可信的推理路径，有效缓解幻觉行为。本工作建立了一个统一的多模态推理忠实性评估与强化框架。代码已开源：https://github.com/lijunxian111/Faithful-First-RPA。
原文摘要 · Abstract (English)
Multimodal Large Language Models (MLLMs) frequently suffer from unfaithfulness, generating reasoning chains that drift from visual evidence or contradict final predictions. We propose Faithful-First Reasoning, Planning, and Acting (RPA) framework in which FaithEvi provides step-wise and chain-level supervision by evaluating the faithfulness of intermediate reasoning, and FaithAct uses these signals to plan and execute faithfulness-aware actions during inference. Experiments across multiple multimodal reasoning benchmarks show that faithful-first RPA improves perceptual faithfulness by up to 24% over prompt-based and tool-augmented reasoning frameworks, without degrading task accuracy. Our analysis shows that treating faithfulness as a guiding principle perceptually faithful reasoning trajectories and mitigates hallucination behavior. This work thereby establishes a unified framework for both evaluating and enforcing faithfulness in multimodal reasoning. Code is at https://github.com/lijunxian111/Faithful-First-RPA.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。