arXiv:2504.05030cs.CVcs.MM2025-04被引 3

建模两人关系的不对称性与时间连续性,提升真实互动场景下的关系识别准确率。

AsyReC: A Multimodal Graph-based Framework for Spatio-Temporal Asymmetric Dyadic Relationship Classification

  • 用双注意力图网络动态加权多模态信息,捕捉关系不对称性
  • 通过片段级建模保持时间连续性,避免采样打断交互过程
  • 引入周期性时间编码,捕捉重复行为模式,适合真实社交场景

二人社交关系由双方反复互动形成,其本质受共同时空体验影响。现有计算方法面临三大挑战:(1) 忽视关系不对称性(如一人视对方为朋友,另一人仅视为熟人);(2) 离散帧采样破坏真实互动的时间连续性;(3) 无法建模周期性行为线索(如规律性发声或重复动作),而这些对推断关系演变至关重要。为此,我们提出 AsyReC——一种基于多模态图神经网络的非对称二元关系分类框架,包含三项创新:(i) 双注意力图网络动态加权多模态信号,捕捉关系不对称性;(ii) 片段级关系学习架构保留时间连续性,实现对现实互动动态的细粒度建模;(iii) 周期性时间编码将时间索引映射至正弦/余弦波形,以建模重复行为模式。在两个公开数据集上的实验表明,该方法达到当前最优性能;消融实验证明,不对称建模与周期性时间编码显著提升模型在真实场景下的鲁棒性。代码已开源:https://github.com/tw-repository/AsyReC。

原文摘要 · Abstract (English)

Dyadic social relationships, which refer to relationships between two individuals who know each other through repeated interactions (or not), are shaped by shared spatial and temporal experiences. Current computational methods for modeling these relationships face three major challenges: (1) the failure to model asymmetric relationships, e.g., one individual may perceive the other as a friend while the other perceives them as an acquaintance, (2) the disruption of continuous interactions by discrete frame sampling, which segments the temporal continuity of interaction in real-world scenarios, and (3) the limitation to consider periodic behavioral cues, such as rhythmic vocalizations or recurrent gestures, which are crucial for inferring the evolution of dyadic relationships. To address these challenges, we propose AsyReC, a multimodal graph-based framework for asymmetric dyadic relationship classification, with three core innovations: (i) a triplet graph neural network with node-edge dual attention that dynamically weights multimodal cues to capture interaction asymmetries (addressing challenge 1); (ii) a clip-level relationship learning architecture that preserves temporal continuity, enabling fine-grained modeling of real-world interaction dynamics (addressing challenge 2); and (iii) a periodic temporal encoder that projects time indices onto sine/cosine waveforms to model recurrent behavioral patterns (addressing challenge 3). Extensive experiments on two public datasets demonstrate state-of-the-art performance, while ablation studies validate the critical role of asymmetric interaction modeling and periodic temporal encoding in improving the robustness of dyadic relationship classification in real-world scenarios. Our code is publicly available at: https://github.com/tw-repository/AsyReC.

关系识别多模态图神经网络时间建模

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。