解决视觉语言模型在动作定位中的语言偏见问题，提升时序推理能力。
Towards Mitigating Modality Bias in Vision-Language Models for Temporal Action Localization
- 通过动态重加权语言模态，仅在语言有优势时才引入
- 在THUMOS14上达到3.2% mAP提升，超越当前最佳方法
- 适合关注多模态融合中公平性与时序建模的研究者
时序动作定位（TAL）需在未剪辑视频中识别动作的边界与类别。尽管视觉语言模型（VLMs）可提供丰富语义以补充视觉证据，但现有方法常过度依赖语言先验，损害视觉性能，导致显著的模态偏见。本文提出ActionVLM，一种系统缓解TAL中模态偏见的视觉语言聚合框架。核心思想是保持视觉为主导信号，仅在语言带来额外收益时才自适应利用语言。为此，我们设计：(i) 去偏重加权模块，估算语言相对于纯视觉预测的增量优势，并动态调整语言权重；(ii) 残差聚合策略，将语言视为互补修正而非主要驱动。该组合有效缓解了模态偏见，降低语言先验带来的过自信，增强时序推理能力。在THUMOS14上的实验表明，模型性能相比最先进方法最高提升3.2% mAP。代码已开源于https://github.com/JiaqiLi404/ActionVLM。
原文摘要 · Abstract (English)
Temporal Action Localization (TAL) requires identifying both the boundaries and categories of actions in untrimmed videos. While vision-language models (VLMs) offer rich semantics to complement visual evidence, existing approaches tend to overemphasize linguistic priors at the expense of visual performance, leading to a pronounced modality bias. We propose ActionVLM, a vision-language aggregation framework that systematically mitigates modality bias in TAL. Our key insight is to preserve vision as the dominant signal while adaptively exploiting language only when beneficial. To this end, we introduce (i) a debiasing reweighting module that estimates the language advantage-the incremental benefit of language over vision-only predictions-and dynamically reweights language modality accordingly, and (ii) a residual aggregation strategy that treats language as a complementary refinement rather than the primary driver. This combination alleviates modality bias, reduces overconfidence from linguistic priors, and strengthens temporal reasoning. Experiments on THUMOS14 show that our model outperforms state-of-the-art by up to 3.2% mAP. Our code is available at https://github.com/JiaqiLi404/ActionVLM
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。