提出新方法提升图像自回归模型压缩效果，让小模型更接近大模型表现。
Knowledge Distillation for Visual Autoregressive Models
- 在学生模型采样基础上进行知识蒸馏，减少视觉标记歧义。
- 在多个图像生成骨干网络上测试，显著缩小与大模型差距。
- 适合需要高效图像生成的部署场景，如移动端或边缘计算。
自回归（AR）图像生成模型表达能力强但计算开销大，亟需有效压缩方法。知识蒸馏（KD）是模型压缩的自然选择，在语言建模中已广泛研究，但在视觉自回归生成领域仍缺乏系统探索。本文首次对视觉自回归模型的蒸馏策略进行系统性分析。结果表明，虽然标准蒸馏可带来一定提升，但为语言模型设计的近期方法难以直接迁移至图像任务：长解码序列和视觉标记歧义导致教师监督在学生条件上下文中不可靠。为此，我们提出VarKD框架，该框架在学生采样基础上进行蒸馏，选择性应用教师监督并降低标记层级歧义。在ImageNet数据集上，对多种AR骨干网络的实验显示，VarKD始终优于现有蒸馏基线，显著缩小与大规模模型的性能差距。
原文摘要 · Abstract (English)
Autoregressive (AR) image generation models are highly expressive but computationally intensive, motivating effective model compression. Knowledge distillation (KD) is a natural approach for model compression and has been widely studied in language modeling, yet its behavior in visual AR generation remains underexplored. In this work, we present the first systematic study of distillation strategies for AR image models. Our analysis shows that while standard distillation can yield meaningful gains, recent methods developed for language do not directly transfer to images: long decoding horizons and visual token ambiguity make teacher supervision unreliable especially under student-conditioned contexts. To address this, we propose VarKD, a distillation framework for visual autoregressive models that distills on student samples while selectively applying teacher supervision and reducing token-level ambiguity. Experiments on ImageNet across multiple AR backbones show that VarKD consistently outperforms prior distillation baselines, narrowing the gap to large-scale models.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。