arXiv:2607.22531cs.CV2026-07

提出孪生表示框架,统一视觉理解与生成的连续特征空间。

Twins: Learn to Predict Unified Representations with Focal Loss

论文配图:Twins: Learn to Predict Unified Representations with Focal Loss
图 1 · 摘自论文原文
  • 将ViT语义特征与VAE低级隐变量通道拼接,构建统一的连续令牌空间。
  • 采用焦点回归优化流匹配,使生成性能提升10.57点gFID(ImageNet)。
  • 适合关注多模态统一表征、生成质量提升的研究者。

统一多模态模型旨在构建一个支持多模态理解与图像生成的共享视觉标记空间。离散方法通过共享码本实现接口统一,而连续方法通常依赖两种分离的表示:用于理解的语义特征(如ViT)和用于合成的低级隐变量(如VAE),导致潜在空间不一致。本文提出Twins,通过在相同标记网格上通道拼接ViT与VAE特征,构建统一的连续标记空间,保持序列长度不变且注意力开销不增加。然而,在扩散变压器中联合建模Twins时出现严重优化失衡:模型能很好拟合ViT部分,却难以匹配VAE隐变量分布。我们追溯其根源为三个异质性来源:频率偏差、内在维度差异以及条件对齐与条件无关的不确定性。为此,我们引入焦点回归目标用于流匹配,对误差较大的VAE维度进行加权,有效平衡了ViT与VAE组件的优化。在ImageNet上,该方法相比朴素MSE损失(无分类器自由引导)取得最高10.57点的gFID提升。Twins在多模态理解基准上表现良好,并提升了重建保真度,缩小了理解与生成导向表示之间的差距。

原文摘要 · Abstract (English)

Unified multimodal models seek a shared visual token space that supports both multimodal understanding and image generation. Discrete methods unify the interface via a shared codebook, whereas continuous pipelines often rely on two disparate representations -- semantic features (e.g., ViT) for understanding and low-level latents (e.g., VAE) for synthesis -- resulting in mismatched latent spaces. We propose Twins, a unified continuous token space formed by channel-wise concatenating ViT and VAE features on the same token grid, so the sequence length is unchanged and attention cost does not increase. However, jointly modeling Twins in a Diffusion Transformer exposes a severe optimization imbalance: the model fits the ViT component well but struggles to match the VAE latent distribution. We trace this imbalance to three sources of heterogeneity: frequency bias, intrinsic dimensionality, and condition-aligned vs condition-independent uncertainty. To address it, we adapt a focal regression objective for flow matching that upweights large-error VAE dimensions, better balancing optimization across the ViT and VAE components. On ImageNet, this yields up to 10.57 gFID gain over naive MSE loss without classifier-free guidance. Twins also performs competitively on multimodal understanding benchmarks and improves reconstruction fidelity, narrowing the gap between understanding- and generation-oriented representations.

多模态生成模型统一表征扩散模型

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。