KVAE系列压缩器让多模态生成更高效，支持音视频图像统一处理。
KVAE: Family of Tokenizers for Multimodal Generative Models
- 基于变分自编码器设计连续压缩模型，支持高保真音视频与图像表示
- 在重建与生成任务中表现优于主流开源模型，如StableAudio、FLUX.2等
- 开源训练细节与代码，适合研究多模态生成的开发者参考使用
潜空间扩散建模（LDM）依赖分词器将输入信号映射为紧凑表示。分词器直接影响生成速度、合成样本质量，并奠定后续应用基础。本文提出一系列面向音频、图像和视频的KVAE分词器：KVAE-Audio为48 kHz全频带连续分词器，潜空间分辨率为50 Hz，通道数64；KVAE-3D为两个因果视频分词器，分别实现4×16×16和4×8×8压缩；KVAE-2D为图像模型，输入压缩比达8倍，通道数32。实验表明，其在重建指标（PSNR、LPIPS、PESQ等）及生成性能（Frechet距离、CLIP分数、CLAP分数等）上达到或超越前沿开源模型，包括Wan-2.2、HunyuanVideo-1.5、FLUX.2、MovieGen、StableAudio与MMAudio。鉴于开发难度，作者公开训练细节、模型选择方法及设计选项消融分析。代码已开源于https://github.com/kandinskylab/kvae 和 https://github.com/kandinskylab/kvae-audio。
原文摘要 · Abstract (English)
Latent diffusion modeling (LDM), a prominent paradigm, utilizes tokenizers to map input signal to compressed representation. This dependency positions tokenizer as an integral part of generation process itself, since it affects learning speed, quality of synthesized samples and lay foundation for later applications. This report presents series of KVAE tokenizers for audio, image and video, all designed for subsequent text-conditioned generation: KVAE-Audio, a continuous full-band 48 kHz tokenizer with a 50 Hz latent of 64 channels; KVAE-3D -- two causal video tokenizers for 4x16x16 and 4x8x8 compression; KVAE-2D, an image model, compressing input by factor of 8 with 32 channels. We demonstrate that reconstruction (PSNR, LPIPS, PESQ, etc.) and generation results on objective (Frechet Distance, CLIP score, CLAP score, etc.) and subjective (side-by-side evaluation) metrics matches or surpasses frontier opensource tokenizers, such as VAEs from Wan-2.2, HunyuanVideo-1.5, FLUX.2, MovieGen, StableAudio and MMAudio. Considering difficulty of development, we share with community training details, model selection method and ablation on design choices. The code is publicly available at https://github.com/kandinskylab/kvae and https://github.com/kandinskylab/kvae-audio.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。