arXiv:2608.04623cs.CV2026-08

用扩散模型融合视觉与文本,实现零样本骨骼动作识别新突破

Visual Anchoring in Diffusion: Multimodal Zero-Shot Skeleton Action Recognition

论文配图:Visual Anchoring in Diffusion: Multimodal Zero-Shot Skeleton Action Recognition
图 1 · 摘自论文原文
  • 通过生成式分类框架,让文本和图像共同指导骨骼特征重建评分
  • 在NTU-120 96/24数据集上达71.3%准确率,超越现有方法
  • 无需测试时微调,适合跨域零样本动作识别场景

零样本骨骼动作识别在未见动作具有相似关节运动但涉及不同物体或场景时仍存在歧义。RGB图像可提供缺失的上下文线索,但现有多模态方法通常独立计算骨骼与RGB的得分并固定融合。由于缺乏无标签测试数据进行适应或融合校准,固定权重无法捕捉类别对之间的模态可靠性差异;而自适应规则又缺少目标侧反馈来判断哪个分支应主导。本文提出规避权重选择问题的生成式分类范式:通过文本条件去噪器预测添加到骨骼特征的噪声,以评估每类的匹配程度。该方法将逐步退化的骨骼特征与固定条件分离,使RGB与文本共同作用于单一分类函数,而非分别输出独立得分。我们提出多模态三元组扩散模型(TDSM-MM),在文本条件去噪Transformer中引入非扩散的RGB条件标记,作为骨骼重建过程中的稳定视觉锚点。大量消融实验表明,TDSM-MM在四个NTU-60/120划分中的三个达到最优归纳准确率,并在NTU-120 96/24上超越当前最先进方法(71.3% vs. 69.1%),且无需测试时适应,证明基于扩散的方法在零样本学习中具有巨大潜力。

原文摘要 · Abstract (English)

Zero-shot Skeleton Action Recognition (ZSAR) remains ambiguous when unseen actions share similar skeleton joint dynamics but differ in objects or scene context. RGB provides these missing cues, yet existing multimodal methods typically maintain independent skeleton and RGB scoring branches and fuse their outputs. Without using unlabeled test data for adaptation or fusion calibration, a fixed fusion weight cannot capture class-pair-dependent modality reliability, while an adaptive rule lacks target-side feedback for deciding which branch should dominate. We bypass this weight-selection problem via the classify-by-generation paradigm, where each class is scored by how accurately a text-conditioned denoiser predicts the noise added to the skeleton feature. This formulation separates the progressively corrupted skeleton from fixed conditioning, allowing RGB and text to jointly condition a single class-scoring function rather than produce independent scores. We instantiate this idea as Multimodal Triplet Diffusion for Skeleton-Text Matching (TDSM-MM), augmenting a text-conditioned denoising Transformer with a non-diffused RGB condition token that serves as a stable visual anchor during skeleton data reconstruction. Our proposed TDSM-MM has been ablated via extensive experiments and achieved the best inductive accuracy on three of four NTU-60/120 splits and surpasses the transductive state-of-the-art on NTU-120 96/24 (i.e., 71.3% vs. 69.1%), without test-time adaptation, suggesting that diffusion-based methods can be a promising direction for zero-shot learning.

零样本识别扩散模型多模态融合

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。