提出轻量级模型与弱监督训练，实现开放词汇动作检测的高效扩展。
Scaling Open-Vocabulary Action Detection
- 采用仅编码器结构，减少参数冗余，提升模型轻量化水平。
- 利用现有封闭集数据进行弱监督预训练，突破数据规模限制。
- 设计新评估基准，避免训练数据泄露，为后续研究提供可靠基线。
本文聚焦开放词汇动作检测的规模化问题。现有方法多局限于封闭集场景，依赖复杂且参数庞大的架构。扩展至开放词汇设置面临两大挑战：（1）缺乏大规模、多类别动作数据集以支持鲁棒训练；（2）对预训练视觉语言对比模型进行参数密集型适配，易导致新增非预训练参数过拟合于基础动作类别。为此，我们提出一种仅编码器的多模态视频动作检测模型，降低对额外参数模块的依赖。其次，设计一种简单弱监督训练策略，利用已有封闭集动作检测数据集进行预训练。最后，摒弃先前工作使用的有缺陷的基类到新类评估基准，构建新评估标准，在不使用封闭集数据进行训练的前提下，基于现有封闭集数据集进行评估，获得可作为未来研究基线的新结果。代码已公开于 https://siatheindochinese.github.io/sia_act_page/。
原文摘要 · Abstract (English)
In this work, we focus on scaling open-vocabulary action detection. Existing approaches for action detection are predominantly limited to closed-set scenarios and rely on complex, parameter-heavy architectures. Extending these models to the open-vocabulary setting poses two key challenges: (1) the lack of large-scale datasets with many action classes for robust training, and (2) parameter-heavy adaptations to a pretrained vision-language contrastive model to convert it for detection, risking overfitting the additional non-pretrained parameters to base action classes. Firstly, we introduce an encoder-only multimodal model for video action detection, reducing the reliance on parameter-heavy additions for video action detection. Secondly, we introduce a simple weakly supervised training strategy to exploit an existing closed-set action detection dataset for pretraining. Finally, we depart from the ill-posed base-to-novel benchmark used by prior works in open-vocabulary action detection and devise a new benchmark to evaluate on existing closed-set action detection datasets without ever using them for training, showing novel results to serve as baselines for future work. Our code is available at https://siatheindochinese.github.io/sia_act_page/ .
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。