提出MVAR框架，用马尔可夫假设降低视觉生成的冗余计算。
MVAR: Visual Autoregressive Modeling with Scale and Spatial Markovian Conditioning
- 引入尺度与空间马尔可夫性，只依赖相邻尺度和局部区域进行预测
- 将注意力复杂度从O(N²)降至O(Nk)，训练仅需8张4090显卡
- 减少3.0倍显存占用，无需缓存键值对，适合高效图像生成
视觉生成的核心在于高效建模视觉数据先验。传统逐标记预测方法通过学习连续标记的条件概率分布来建模；近期方法则转向多尺度表示的逐尺度预测，显著降低生成延迟。然而，这些方法在每一尺度上依赖所有先前尺度，并要求每个标记关注所有历史标记，存在尺度与空间冗余。为此，我们提出马尔可夫视觉自回归模型（MVAR），通过引入尺度与空间马尔可夫假设，缓解冗余问题。具体而言，设计仅依赖相邻前一尺度特征的尺度-马尔可夫轨迹，支持并行训练，大幅降低显存消耗。同时提出空间-马尔可夫注意力，限制每个标记仅关注相邻尺度上对应位置的局部邻域（大小为k），而非跨尺度全部标记。基于此改进，注意力计算复杂度由O(N²)降至O(Nk)，实现仅用八张NVIDIA RTX 4090 GPU完成训练，且推理无需KV缓存。ImageNet上的大量实验表明，MVAR在小模型从头训练和大模型微调场景下均达到相当或更优性能，平均显存占用降低3.0倍。
原文摘要 · Abstract (English)
Essential to visual generation is efficient modeling of visual data priors. Conventional next-token prediction methods define the process as learning the conditional probability distribution of successive tokens. Recently, next-scale prediction methods redefine the process to learn the distribution over multi-scale representations, significantly reducing generation latency. However, these methods condition each scale on all previous scales and require each token to consider all preceding tokens, exhibiting scale and spatial redundancy. To better model the distribution by mitigating redundancy, we propose Markovian Visual AutoRegressive modeling (MVAR), a novel autoregressive framework that introduces scale and spatial Markov assumptions to reduce the complexity of conditional probability modeling. Specifically, we introduce a scale-Markov trajectory that only takes as input the features of adjacent preceding scale for next-scale prediction, enabling the adoption of a parallel training strategy that significantly reduces GPU memory consumption. Furthermore, we propose spatial-Markov attention, which restricts the attention of each token to a localized neighborhood of size k at corresponding positions on adjacent scales, rather than attending to every token across these scales, for the pursuit of reduced modeling complexity. Building on these improvements, we reduce the computational complexity of attention calculation from O(N^2) to O(Nk), enabling training with just eight NVIDIA RTX 4090 GPUs and eliminating the need for KV cache during inference. Extensive experiments on ImageNet demonstrate that MVAR achieves comparable or superior performance with both small model trained from scratch and large fine-tuned models, while reducing the average GPU memory footprint by 3.0x.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。