用统一模型实现无需标签的音视频说话人验证，更高效且抗模态缺失。
SSAVSV: Towards Unified Model for Self-Supervised Audio-Visual Speaker Verification
- 设计统一框架，共享视觉变压器主干处理音视频输入。
- 自监督对比学习+不对称掩码，在无标签数据下性能媲美有监督方法。
- 计算成本低，支持单模态或跨模态输入，适合实际部署。
传统音视频说话人验证依赖大量标注数据并采用独立的模态专用架构，计算开销大，难以扩展。为解决此问题，我们提出一种基于对比学习、不对称掩码与掩码数据建模的自监督学习框架，以获得鲁棒的音视频特征表示。特别地，我们采用统一框架，使用单一共享的视觉变换器主干处理音频、视觉或音视频输入，在训练和测试阶段均保持高效，并具备对缺失模态的鲁棒性。大量实验表明，该方法在无标签数据条件下达到竞争性性能，同时相比传统方法显著降低计算成本。
原文摘要 · Abstract (English)
Conventional audio-visual methods for speaker verification rely on large amounts of labeled data and separate modality-specific architectures, which is computationally expensive, limiting their scalability. To address these problems, we propose a self-supervised learning framework based on contrastive learning with asymmetric masking and masked data modeling to obtain robust audiovisual feature representations. In particular, we employ a unified framework for self-supervised audiovisual speaker verification using a single shared backbone for audio and visual inputs, leveraging the versatility of vision transformers. The proposed unified framework can handle audio, visual, or audiovisual inputs using a single shared vision transformer backbone during training and testing while being computationally efficient and robust to missing modalities. Extensive experiments demonstrate that our method achieves competitive performance without labeled data while reducing computational costs compared to traditional approaches.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。