发现并修复大模型多模态感知与决策脱节问题。
Diagnosing and Mitigating Perception-Decision Misalignment in Omni-LLMs via Modality Subspace Activation
- 用双视角分析感知-决策偏差，量化模态敏感性。
- 实测主流模型移除模态后仍无响应，敏感性极低。
- 无需训练，在推理时动态调整模态权重，提升感知一致性。
Omni-Large Language Models (Omni-LLMs) 在世界动作模型和自主代理等复杂多模态推理任务中发挥关键作用。然而，其优异表现常掩盖深层的感知-决策偏差（Perceptual-Decision Misalignment, PDM），即决策未能忠实反映多模态输入。为诊断该问题，我们提出因果模态敏感性（Causal Modality Sensitivity, CMS），通过双视角框架实现：宏观行为层面的答保留率（Answer Retention Rate, ARR），以及微观分布偏移的对数角度差异（Logit Angular Discrepancy, LAD）。我们构建了因果模态基准数据集 CausalMSBench，专门隔离语言先验影响。基准测试显示，主流 Omni-LLMs 的 CMS 极低，即使关键模态被移除，其输出分布变化亦可忽略不计。为此，我们提出模态子空间激活（Modality Subspace Activation, MSA）——一种无需训练的推理时框架，利用奇异值分解（SVD）估算各模态激活强度，动态平衡最后一层隐藏状态中的模态投影，有效恢复多个基准上的 CMS。
原文摘要 · Abstract (English)
Omni-Large Language Models (Omni-LLMs) power complex multi-modal reasoning in applications like World Action Models and autonomous agents. However, their strong performance often masks a profound Perceptual-Decision Misalignment (PDM), where decisions remain unfaithful to multi-modal perceptions. To diagnose this, we formalize Causal Modality Sensitivity (CMS), operationalized via a dual-lens framework: Answer Retention Rate (ARR) at the macro behavioral level, and Logit Angular Discrepancy (LAD) to track microscopic distribution shifts. We also curate CausalMSBench, a diagnostic dataset isolating language priors. Benchmarking reveals that popular Omni-LLMs exhibit critically low CMS, showing negligible distribution shifts even when key modalities are removed. To rectify this, we propose Modality Subspace Activation (MSA), a training-free inference-time framework that uses Singular Value Decomposition (SVD) to estimate modal activation strengths. MSA dynamically balances modal projections in the last hidden state, effectively restoring CMS across benchmarks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。