用动态约束提升多模型推理在变化环境中的鲁棒性
Turning Drift into Constraint: Robust Reasoning Alignment in Non-Stationary Multi-Stream Environments
- 将模型分歧视为动态负约束，构建约束优化框架
- 7B模型在胸片诊断中超越源模型平均准确率
- 适用于医疗多模态模型对齐与抗漂移研究
本文揭示了多模态大语言模型（MLLMs）在非平稳环境中推理对齐的关键挑战：源模型的推理分布会不可预测地演化，导致系统性偏差与漂移传递至目标模型。为此，我们基于概念漂移理论，将多源推理对齐建模为约束满足问题。提出自主偏好优化（APO）框架，将模型间差异视为动态负约束，而非噪声。APO采用两阶段协议：首先通过监督微调将目标模型投影至源模型能力并集；其次通过多负向Plackett-Luce目标显式抑制漂移轨迹，合成一致共识流形。在胸片解读任务上的实验表明，我们的7B模型实现了更优鲁棒性，平均准确率超过多个专有源模型。此外，我们发布CXR-MAX，一个包含7个大规模MLLM生成的170,982条推理轨迹的大规模基准数据集，以推动漂移环境下推理对齐研究。代码与数据已开源。
原文摘要 · Abstract (English)
This paper identifies a critical yet underexplored challenge in reasoning alignment from multiple multi-modal large language models (MLLMs): In non-stationary environments, the diverse reasoning distributions of source models often evolve unpredictably, transmitting systematic biases and drift to the target model. To address this, we formulate multi-source reasoning alignment as a constraint satisfaction problem under concept drift theory. We propose Autonomous Preference Optimization (APO), a novel framework that treats inter-model divergences not as noise, but as dynamic negative constraints. APO operates via a two-stage protocol: first, supervised bootstrapping projects the target model into the capability union of source models; second, constraint-aware optimization synthesizes a consistent consensus manifold by explicitly suppressing drifting trajectories via a multi-negative Plackett-Luce objective. Extensive experiments on chest X-ray interpretation demonstrate that our 7B model achieves superior robustness, outperforming even proprietary source models in average accuracy. Furthermore, we release CXR-MAX, a large-scale benchmark comprising 170,982 reasoning trajectories from seven large-scale MLLMs to facilitate research on reasoning alignment under drift. Code and data are available at: https://github.com/XiaoyuYoung/APO.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。