让视频语言模型学会理解动作概念，提升对未见过动作的识别能力。
ACE: Action Concept Enhancement of Video-Language Models in Procedural Videos
- 用随机替换标签的方式，在训练中引入动作同义词和反例。
- 在烹饪与组装场景中，零样本识别准确率显著提升。
- 适合需要理解动作本质而非死记硬背标签的应用场景。
视觉语言模型（VLM）具备识别未见动作的能力，但缺乏对程序性动作概念的内在理解，容易过度依赖固定标签，对未见过的动作同义词不具鲁棒性。为此，我们提出一种简单微调方法——动作概念增强（ACE），通过在辅助分类损失中随机替换固定标签，持续引入增强后的动作同义词与负样本，使模型在微调过程中生成新的动作标签组合，避免对固定动作表示的过拟合。通过可视化嵌入空间中未见动作同义词的对齐情况，验证了模型概念理解能力的提升。在ATA、IKEA和GTEA数据集上的实验表明，该方法在烹饪与组装领域显著提升了零样本动作分类性能，同时保持对已见动作的竞争力。
原文摘要 · Abstract (English)
Vision-language models (VLMs) are capable of recognizing unseen actions. However, existing VLMs lack intrinsic understanding of procedural action concepts. Hence, they overfit to fixed labels and are not invariant to unseen action synonyms. To address this, we propose a simple fine-tuning technique, Action Concept Enhancement (ACE), to improve the robustness and concept understanding of VLMs in procedural action classification. ACE continually incorporates augmented action synonyms and negatives in an auxiliary classification loss by stochastically replacing fixed labels during training. This creates new combinations of action labels over the course of fine-tuning and prevents overfitting to fixed action representations. We show the enhanced concept understanding of our VLM, by visualizing the alignment of encoded embeddings of unseen action synonyms in the embedding space. Our experiments on the ATA, IKEA and GTEA datasets demonstrate the efficacy of ACE in domains of cooking and assembly leading to significant improvements in zero-shot action classification while maintaining competitive performance on seen actions.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。