发现大模型有偏好模态的倾向，并可人为调控以提升性能。
Evaluating and Steering Modality Preferences in Multimodal Large Language Model
- 构建冲突场景评测工具MC²，系统检测多模态模型偏好。
- 20个模型均显偏好，且偏好程度与任务表现相关。
- 无需微调即可通过表示工程引导偏好，提升推理能力。
多模态大语言模型（MLLM）在复杂多模态任务中取得显著进展，但其是否表现出模态偏好——即在处理多模态信息时倾向于某一模态——仍缺乏深入研究。为此，我们提出MC²基准，通过构建受控的证据冲突场景，系统评估模型在决策中的模态偏好。大量实验表明，所有20个测试的MLLM均表现出明显的模态偏好，且该偏好可作为下游任务性能的有用预测指标。进一步分析显示，模态偏好可通过指令引导进行控制，并存在于模型的潜在表示中。基于此，我们提出一种基于表示工程的探测与调控方法，无需额外微调即可显式控制模态偏好，有效增强模型向目标方向的偏好，并在多个多模态理解与推理任务中展现显著性能提升。
原文摘要 · Abstract (English)
Multi-modal large language models (MLLMs) have achieved remarkable success on complex multi-modal tasks. However, it remains insufficiently explored whether they exhibit $\textbf{modality preference}$, a tendency to favor one modality over another when processing multi-modal contexts. To study this question, we introduce $\textbf{MC\textsuperscript{2}}$ benchmark, which constructs controlled evidence-conflict scenarios to systematically evaluate modality preference in decision-making. Extensive experiments reveal that all 20 tested MLLMs generally demonstrate clear modality preferences, and such preferences can serve as a useful indicator of downstream task performance of MLLMs. Further analysis shows that modality preference can be controlled by instruction guidance and captured within the latent representations of MLLMs. Built on these insights, we propose a probing and steering method based on representation engineering to explicitly control modality preference without requiring additional fine-tuning. This method effectively amplifies modality preference toward a desired direction and demonstrates promising improvements across multiple multi-modal understanding and reasoning tasks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。