通过结构化视觉令牌空间,大幅提升自回归图像生成效率与质量。
MASC: Boosting Autoregressive Image Generation with a Manifold-Aligned Semantic Clustering
- 基于嵌入流形构建分层语义树,优化预测空间结构。
- 训练速度提升57%,LlamaGen-XL的FID从2.87降至2.58。
- 可直接接入现有模型,适合追求高效高质量生成的研究者。
自回归(AR)图像生成模型虽具潜力,但其核心组件——庞大且无结构的视觉令牌词表——导致根本性低效。传统方法将令牌视为平坦词汇,忽略嵌入空间中邻近性与语义相似性的内在关联,使预测任务过于复杂,影响训练效率与生成质量。为此,我们提出流形对齐语义聚类(MASC),一种基于代码本内在结构构建分层语义树的原理性框架。MASC采用新型几何感知距离度量与密度驱动的凝聚式构建方法,建模令牌嵌入的底层流形。通过将高维平坦预测任务转化为结构化的层级任务,引入有益归纳偏置,显著简化学习问题。MASC为即插即用模块,实验验证其有效性:训练速度最高提升57%,生成质量显著改善,LlamaGen-XL的FID从2.87降至2.58。MASC使现有AR框架达到与顶尖方法相当的性能,表明结构化预测空间与架构创新同样关键。
原文摘要 · Abstract (English)
Autoregressive (AR) models have shown great promise in image generation, yet they face a fundamental inefficiency stemming from their core component: a vast, unstructured vocabulary of visual tokens. This conventional approach treats tokens as a flat vocabulary, disregarding the intrinsic structure of the token embedding space where proximity often correlates with semantic similarity. This oversight results in a highly complex prediction task, which hinders training efficiency and limits final generation quality. To resolve this, we propose Manifold-Aligned Semantic Clustering (MASC), a principled framework that constructs a hierarchical semantic tree directly from the codebook's intrinsic structure. MASC employs a novel geometry-aware distance metric and a density-driven agglomerative construction to model the underlying manifold of the token embeddings. By transforming the flat, high-dimensional prediction task into a structured, hierarchical one, MASC introduces a beneficial inductive bias that significantly simplifies the learning problem for the AR model. MASC is designed as a plug-and-play module, and our extensive experiments validate its effectiveness: it accelerates training by up to 57% and significantly improves generation quality, reducing the FID of LlamaGen-XL from 2.87 to 2.58. MASC elevates existing AR frameworks to be highly competitive with state-of-the-art methods, establishing that structuring the prediction space is as crucial as architectural innovation for scalable generative modeling.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。