用音频压缩模型生成动态视觉，让声音直接驱动画面变化。
LAV: Audio-Driven Dynamic Visual Generation with Neural Compression and StyleGAN2
- 用音频编码器嵌入直接转为图像风格潜空间
- 无需显式特征映射，实现语义连贯的音画转换
- 适合音乐可视化、艺术创作等创意应用
本文提出LAV（Latent Audio-Visual），将EnCodec神经音频压缩与StyleGAN2生成能力结合，实现由预录制音频驱动的动态视觉生成。与以往依赖显式特征映射的方法不同，LAV采用EnCodec嵌入作为潜在表示，通过随机初始化的线性映射直接转换至StyleGAN2的风格潜空间。该方法保留了转换过程中的语义丰富性，实现了细腻且语义连贯的音画对应。框架展示了预训练音频压缩模型在艺术与计算应用中的潜力。
原文摘要 · Abstract (English)
This paper introduces LAV (Latent Audio-Visual), a system that integrates EnCodec's neural audio compression with StyleGAN2's generative capabilities to produce visually dynamic outputs driven by pre-recorded audio. Unlike previous works that rely on explicit feature mappings, LAV uses EnCodec embeddings as latent representations, directly transformed into StyleGAN2's style latent space via randomly initialized linear mapping. This approach preserves semantic richness in the transformation, enabling nuanced and semantically coherent audio-visual translations. The framework demonstrates the potential of using pretrained audio compression models for artistic and computational applications.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。