提出解耦式音视频数据蒸馏方法,提升压缩后数据质量
Decoupled Audio-Visual Dataset Distillation
- 用预训练库稳定提取音视频特征,通过轻量解耦器分离共性与私有表示
- 在不同样本量下均优于现有方法,最高提升2.3%准确率
- 适合需要高质量音视频数据压缩的研究者使用
音视频数据蒸馏旨在将大规模数据集压缩为紧凑子集,同时保持原始性能。然而,传统分布匹配方法难以捕捉跨模态内在对齐。后续研究虽引入跨模态匹配,但仍面临两大挑战:(i) 独立随机初始化编码器导致模态映射空间不一致,增加训练难度;(ii) 模态间直接交互会损害模态特有(私有)信息,降低蒸馏数据质量。为此,我们提出基于预训练的解耦音视频蒸馏框架DAVDD。DAVDD利用多样化预训练库获取稳定模态特征,并通过轻量解耦器库将其分解为共性与私有表示。为有效保留跨模态结构,进一步引入共性跨模态匹配与样本-分布联合对齐策略,确保共享表示在样本级和全局分布级均对齐。同时,私有表示完全隔离于跨模态交互,全程保护模态特有线索。多基准测试表明,DAVDD在所有每类样本数(IPC)设置下均达到当前最优,验证了解耦表示学习在高质量音视频数据蒸馏中的有效性。代码将公开。
原文摘要 · Abstract (English)
Audio-Visual Dataset Distillation aims to compress large-scale datasets into compact subsets while preserving the performance of the original data. However, conventional Distribution Matching (DM) methods struggle to capture intrinsic cross-modal alignment. Subsequent studies have attempted to introduce cross-modal matching, but two major challenges remain: (i) independently and randomly initialized encoders lead to inconsistent modality mapping spaces, increasing training difficulty; and (ii) direct interactions between modalities tend to damage modality-specific (private) information, thereby degrading the quality of the distilled data. To address these challenges, we propose DAVDD, a pretraining-based decoupled audio-visual distillation framework. DAVDD leverages a diverse pretrained bank to obtain stable modality features and uses a lightweight decoupler bank to disentangle them into common and private representations. To effectively preserve cross-modal structure, we further introduce Common Intermodal Matching together with a Sample-Distribution Joint Alignment strategy, ensuring that shared representations are aligned both at the sample level and the global distribution level. Meanwhile, private representations are entirely isolated from cross-modal interaction, safeguarding modality-specific cues throughout distillation. Extensive experiments across multiple benchmarks show that DAVDD achieves state-of-the-art results under all IPC settings, demonstrating the effectiveness of decoupled representation learning for high-quality audio-visual dataset distillation. Code will be released.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。