轻量级模型实现用户自定义动作的低资源持续学习,保护隐私不存数据。
POET: Prompt Offset Tuning for Continual Human Action Adaptation

- 用可学习的时空提示偏移微调轻量骨干网络,无需预训练大模型。
- 在少样本下持续添加新动作类别,性能优于主流基线方法。
- 适用于隐私敏感场景,适合开发可个性化交互的沉浸式设备。
随着扩展现实(XR)重塑人机交互方式,人体动作识别研究日益重要。当前部署在沉浸式设备上的模型通常固定且仅支持预设动作类别。本研究旨在让用户和开发者能以低资源、高效的方式持续添加新动作类别,且过程无需存储或回放用户的敏感训练数据。我们提出隐私感知的少样本持续动作识别问题,并引入POET:提示偏移微调方法。不同于现有提示微调需依赖大型预训练模型,POET在仅基于基础类别数据预训练的轻量级骨干网络上实现了有效微调,首次将此类提示微调应用于图神经网络。我们构建了两个新基准:(i) NTU RGB+D用于动作识别,(ii) SHREC-2017用于手势识别。实验表明,POET在各项指标上均显著优于多种基线方法。代码开源于https://github.com/humansensinglab/POET-continual-action-recognition。
原文摘要 · Abstract (English)
As extended reality (XR) is redefining how users interact with computing devices, research in human action recognition is gaining prominence. Typically, models deployed on immersive computing devices are static and limited to their default set of classes. The goal of our research is to provide users and developers with the capability to personalize their experience by adding new action classes to their device models continually. Importantly, a user should be able to add new classes in a low-shot and efficient manner, while this process should not require storing or replaying any of user's sensitive training data. We formalize this problem as privacy-aware few-shot continual action recognition. Towards this end, we propose POET: Prompt-Offset Tuning. While existing prompt tuning approaches have shown great promise for continual learning of image, text, and video modalities; they demand access to extensively pretrained transformers. Breaking away from this assumption, POET demonstrates the efficacy of prompt tuning a significantly lightweight backbone, pretrained exclusively on the base class data. We propose a novel spatio-temporal learnable prompt offset tuning approach, and are the first to apply such prompt tuning to Graph Neural Networks. We contribute two new benchmarks for our new problem setting in human action recognition: (i) NTU RGB+D dataset for activity recognition, and (ii) SHREC-2017 dataset for hand gesture recognition. We find that POET consistently outperforms comprehensive benchmarks. Source code at https://github.com/humansensinglab/POET-continual-action-recognition.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。