arXiv:2510.21093cs.AI2025-10

医学视觉问答新框架,让AI回答更准更省力。

MedAlign: A Synergistic Framework of Multimodal Preference Optimization and Federated Meta-Cognitive Reasoning

  • 用多模态偏好优化对齐视觉内容与答案
  • 减少幻觉,推理长度缩短51.6%,准确率提升11.85%
  • 适合医疗AI研发与跨机构协作场景

大模型在智能医疗中潜力巨大,但其在临床服务中的应用受限于三大挑战:生成答案脱离视觉证据、固定深度推理效率低、多机构协作困难。为此,本文提出MedAlign框架,用于提升医学视觉问答(Med-VQA)的视觉准确性。首先,设计多模态直接偏好优化(mDPO)目标,显式对齐偏好学习与视觉上下文;其次,构建检索感知的专家混合(RA-MoE)架构,通过图像与文本相似度路由查询至特定专家模型,缓解幻觉问题;最后,引入联邦治理机制,由本地微调的专家基于mDPO执行迭代思维链(CoT)推理,并结合元认知不确定性估计实现自适应推理。在三个代表性Med-VQA数据集上的实验表明,MedAlign性能达到当前最优,相比强基线模型F1得分最高提升11.85%,同时推理平均长度比固定深度方法减少51.60%。

原文摘要 · Abstract (English)

Recently, large models have shown significant potential for smart healthcare. However, the deployment of Large Vision-Language Models (LVLMs) for clinical services is currently hindered by three critical challenges: a tendency to hallucinate answers not grounded in visual evidence, the inefficiency of fixed-depth reasoning, and the difficulty of multi-institutional collaboration. To address these challenges, in this paper, we develop MedAlign, a novel framework to ensure visually accurate LVLM responses for Medical Visual Question Answering (Med-VQA). Specifically, we first propose a multimodal Direct Preference Optimization (mDPO) objective to explicitly align preference learning with visual context. We then design a Retrieval-Aware Mixture-of-Experts (RA-MoE) architecture that utilizes image and text similarity to route queries to a specialized and context-augmented LVLM (i.e., an expert), thereby mitigating hallucinations in LVLMs. To achieve adaptive reasoning and facilitate multi-institutional collaboration, we propose a federated governance mechanism, where the selected expert, fine-tuned on clinical datasets based on mDPO, locally performs iterative Chain-of-Thought (CoT) reasoning via the local meta-cognitive uncertainty estimator. Extensive experiments on three representative Med-VQA datasets demonstrate that MedAlign achieves state-of-the-art performance, outperforming strong retrieval-augmented baselines by up to $11.85\%$ in F1-score, and simultaneously reducing the average reasoning length by $51.60\%$ compared with fixed-depth CoT approaches.

医学AI视觉问答多模态联邦学习

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。