arXiv:2608.05450cs.CV2026-08

MOSAIK按区域精细分配图像补丁,高效生成同时保持高画质。

MOSAIK: Multi-Patch Content-Aware Spatial Allocation of Image Tokens for Efficient Generation

论文配图:MOSAIK: Multi-Patch Content-Aware Spatial Allocation of Image Tokens for Efficient Generation
图 1 · 摘自论文原文
  • 根据区域敏感度动态调整补丁大小,避免统一粗粒度处理
  • 降低70%计算量、83%令牌数,仅损失1.0分评测得分
  • 适合资源受限下需高画质生成的场景,如移动端部署

像素空间扩散模型直接在图像空间生成,突破了潜在空间模型的重建瓶颈,但其高令牌数量导致自注意力计算复杂度呈二次增长。现有方法通过在特定去噪步骤使用更大补丁来降低计算成本,但所有区域均采用统一补丁大小,忽略了不同区域在粗化时的保真度损失差异。本文提出MOSAIK,一种基于损伤引导的框架,实现跨区域与去噪步的异构补丁布局。MOSAIK适配PixelDiT主干网络以生成任意非均匀补丁结构,并引入轻量级预测器,利用中间去噪特征估计各区域因粗化造成的保真度损失。在给定令牌预算下,损伤引导布局预测器将细粒度补丁分配至敏感区域,其余区域使用粗粒度补丁。实验显示,相较全计算量的PixelDiT,MOSAIK降低70%计算量(FLOPs)、83%令牌数,且在GenEval上表现相当,DPG-Bench得分仅下降1.0点。相比时间补丁调度、特征缓存等效率范式,在中等预算下性能更优,且在极端计算约束下持续领先基线。

原文摘要 · Abstract (English)

Pixel-space diffusion models avoid the reconstruction ceiling of latent diffusion models by generating directly in image space. However, their substantially higher token count makes generation expensive due to the quadratic complexity of self-attention. Several existing efficiency methods reduce this cost by using larger patches at selected denoising steps, thereby representing the image with fewer tokens. Yet, each step still uses a single patch size uniformly across the entire image, overlooking that different regions suffer different fidelity losses when coarsened. We introduce MOSAIK, a damage-guided framework that varies patch size across regions and denoising steps. MOSAIK adapts the PixelDiT backbone to generate arbitrary heterogeneous patch layouts, and a lightweight predictor uses intermediate denoising features to estimate the fidelity loss caused by coarsening each region. Given a token budget, our damage-guided layout predictor assigns fine patches to sensitive regions and coarse patches elsewhere. Remarkably, while reducing FLOPs by 70% and token count by 83%, MOSAIK matches the full-compute PixelDiT on GenEval and its DPG-Bench score drops by only 1.0 point. Compared to diverse efficiency paradigms, including temporal patch scheduling and feature caching, our approach delivers highly competitive performance at moderate budgets and consistently outperforms these baselines in highly constrained compute regimes.

图像生成扩散模型高效推理补丁布局

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。