解决视觉分词中重建与语义的冲突，提升生成质量与感知能力。
MUSE: Resolving Manifold Misalignment in Visual Tokenization via Topological Orthogonality
- 通过拓扑正交性解耦结构与语义优化，避免梯度冲突。
- 生成质量达gFID 3.08，线性探测准确率85.2%，超越教师模型。
- 适合追求高质量图像生成与鲁棒表征学习的研究者。
统一视觉分词面临高保真像素重建（空间等变性）与语义抽象（概念不变性）之间的根本权衡。我们将其归因于流形错位：联合优化导致相反梯度，形成零和博弈。为此，提出基于拓扑正交性的MUSE框架。通过将结构视为正交桥梁，MUSE在Transformer内解耦优化：结构梯度调整注意力拓扑，语义梯度更新特征值。此设计将破坏性干扰转为相互增强。实验表明，MUSE打破该权衡，生成质量达gFID 3.08，线性探测准确率85.2%（优于教师InternViT-300M的82.5%），证明结构对齐的重建可增强语义感知。代码见https://github.com/PanqiYang1/MUSE。
原文摘要 · Abstract (English)
Unified visual tokenization faces a fundamental trade-off between high-fidelity pixel reconstruction (spatial equivariance) and semantic abstraction (conceptual invariance). We attribute this conflict to Manifold Misalignment: naive joint optimization induces opposing gradients, creating a zero-sum game between reconstruction and perception. To address this, we propose MUSE, a framework based on Topological Orthogonality. By treating Structure as an orthogonal bridge, MUSE decouples optimization within Transformers: structural gradients refine attention topology, while semantic gradients update feature values. This turns destructive interference into Mutual Reinforcement. Experiments show that MUSE breaks the trade-off, achieving state-of-the-art generation quality (gFID 3.08) and surpassing its teacher InternViT-300M in linear probing (85.2\% vs. 82.5\%), demonstrating that structurally aligned reconstruction can enhance semantic perception. Code is available at https://github.com/PanqiYang1/MUSE.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。