无需目标数据,测试时在线适应视角差异,提升动作预测准确率。
Test-time Ego-Exo-centric Adaptation for Action Anticipation via Multi-Label Prototype Growing and Dual-Clue Consistency
- 通过多标签原型生长机制动态更新记忆库,平衡多个动作类别。
- 引入文本与视觉双线索一致性,缓解视角间时空差异导致的性能下降。
- 适用于人机协作等场景,尤其适合无目标数据标注的实时应用。
在人机协作等场景中,自拍视角(Ego)与外拍视角(Exo)间的高效适配至关重要。然而,现有方法大多依赖目标视角数据进行训练,增加了计算与数据采集成本。本文首次提出测试时视角自适应动作预测(TE$^{2}$A$^{3}$)任务,旨在测试阶段对源视角训练好的模型进行在线调整,以预测目标视角动作。该任务面临多动作候选与显著时空视差的挑战,现有测试时自适应方法难以应对。为此,我们提出双线索增强原型生长网络(DCPGN),通过多标签原型生长模块(ML-PGM)实现多标签分配与置信度重加权,构建类级记忆库,并采用熵优先队列策略更新;同时设计双线索一致性模块(DCCM),引入轻量级叙述器生成描述动作进展的文本线索,与包含各类物体的视觉线索互补,约束推理出的文本与视觉逻辑一致性,实现时空跨视角对齐。在新提出的EgoMe-anti及现有EgoExoLearn基准上的大量实验表明,本方法显著优于当前最优方法。代码已开源。
原文摘要 · Abstract (English)
Efficient adaptation between Egocentric (Ego) and Exocentric (Exo) views is crucial for applications such as human-robot cooperation. However, the success of most existing Ego-Exo adaptation methods relies heavily on target-view data for training, thereby increasing computational and data collection costs. In this paper, we make the first exploration of a Test-time Ego-Exo Adaptation for Action Anticipation (TE$^{2}$A$^{3}$) task, which aims to adjust the source-view-trained model online during test time to anticipate target-view actions. It is challenging for existing Test-Time Adaptation (TTA) methods to address this task due to the multi-action candidates and significant temporal-spatial inter-view gap. Hence, we propose a novel Dual-Clue enhanced Prototype Growing Network (DCPGN), which accumulates multi-label knowledge and integrates cross-modality clues for effective test-time Ego-Exo adaptation and action anticipation. Specifically, we propose a Multi-Label Prototype Growing Module (ML-PGM) to balance multiple positive classes via multi-label assignment and confidence-based reweighting for class-wise memory banks, which are updated by an entropy priority queue strategy. Then, the Dual-Clue Consistency Module (DCCM) introduces a lightweight narrator to generate textual clues indicating action progressions, which complement the visual clues containing various objects. Moreover, we constrain the inferred textual and visual logits to construct dual-clue consistency for temporally and spatially bridging Ego and Exo views. Extensive experiments on the newly proposed EgoMe-anti and the existing EgoExoLearn benchmarks show the effectiveness of our method, which outperforms related state-of-the-art methods by a large margin. Code is available at \href{https://github.com/ZhaofengSHI/DCPGN}{https://github.com/ZhaofengSHI/DCPGN}.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。