用扩散模型提取特征，实现跨物种视角的强动作识别
Diffusion-Based Action Recognition Generalizes to Untrained Domains
- 用扩散模型在早期时间步提取语义特征，弱化像素细节干扰
- 在跨物种、跨视角、跨场景三类测试中均刷新最优性能
- 适合需要强泛化能力的动作识别研究者使用
人类能在巨大上下文和视角差异下识别相同动作，如蜘蛛与马的行走、第一人称与第三人称视角、真实场景与电影画面。当前深度学习模型在这些条件下表现不佳。本文提出利用视觉扩散模型（VDM）生成特征，并通过变压器聚合，实现跨条件的人类级动作识别。实验发现，在扩散过程早期时间步条件化模型可增强语义信息、抑制像素级细节。我们在跨物种、跨视角、跨记录场景三类泛化任务中验证方法，模型在所有基准上均达到新最优，使机器动作识别更接近人类鲁棒性。项目页：https://www.vision.caltech.edu/actiondiff。代码：https://github.com/frankyaoxiao/ActionDiff。
原文摘要 · Abstract (English)
Humans can recognize the same actions despite large context and viewpoint variations, such as differences between species (walking in spiders vs. horses), viewpoints (egocentric vs. third-person), and contexts (real life vs movies). Current deep learning models struggle with such generalization. We propose using features generated by a Vision Diffusion Model (VDM), aggregated via a transformer, to achieve human-like action recognition across these challenging conditions. We find that generalization is enhanced by the use of a model conditioned on earlier timesteps of the diffusion process to highlight semantic information over pixel level details in the extracted features. We experimentally explore the generalization properties of our approach in classifying actions across animal species, across different viewing angles, and different recording contexts. Our model sets a new state-of-the-art across all three generalization benchmarks, bringing machine action recognition closer to human-like robustness. Project page: https://www.vision.caltech.edu/actiondiff. Code: https://github.com/frankyaoxiao/ActionDiff
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。