用生成潜码直接增强视觉模型,突破跨模态蒸馏效率瓶颈。
Synesthesia via Direct Latent Augmentation:Bypassing the Decode-Encode Loop for Cross-Modal Distillation

- 跳过解码重编码循环,直接使用生成潜码作为额外信息
- 在无配对数据下实现比传统方法更高的分类准确率
- 适合资源受限场景下的高效多模态知识迁移
多模态融合虽能显著提升计算机视觉模型性能,但部署成本高昂且依赖稀缺的完全配对数据。现有方法通过生成AI合成缺失模态以缓解数据瓶颈,却引入严重低效:解码-编码循环。生成潜码被解码为噪声原始信号,下游分类器需重新编码,浪费计算资源。为此,我们提出直接潜码增强(DLA),直接使用未解码的生成潜码作为优势信息。为进一步将密集知识传递给纯视觉学生模型,提出多层显式模拟联觉(MESSy)。不同于强制严格表示匹配导致学生特征扭曲,MESSy采用预测目标,安全内化物理先验。实验表明,本框架显著优于原始数据增强与传统蒸馏。最终,所提方法训练出高精度单模态学生模型,其潜空间具备从未直接观察过的模态的“联觉”结构。
原文摘要 · Abstract (English)
While multimodal integration significantly improves computer vision models, deploying them incurs prohibitive inference costs and requires scarce, perfectly paired datasets. Recent methods address this data bottleneck by synthesizing missing modalities via generative AI, yet they introduce a severe inefficiency: the Decode-Encode Loop. Specifically, information-rich generative latents are decoded into noisy raw signals, forcing the downstream classifier to waste capacity re-encoding them. To bypass this bottleneck, we propose Direct Latent Augmentation (DLA), utilizing undecoded generative latents directly as privileged information. Furthermore, to transfer this dense knowledge to a purely visual student, we introduce Multilayer Explicit Simulated Synesthesia (MESSy). Instead of enforcing rigid representation matching, which forces the student to distort its native visual features to accommodate complex multimodal topologies, MESSy uses a predictive objective to safely internalize these physical priors. Empirical results demonstrate that our framework significantly outperforms raw data augmentation and traditional distillation. Ultimately, our approach yields highly accurate unimodal students with "synesthetic" latent structures that are inherently aligned with modalities they have never directly observed.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。