通过自适应压缩高低频图像区域,显著提升生成式图像编辑速度。
HiLo-Token: Input-Adaptive High-Low Frequency Token Compression for Efficient Image Editing

- 根据用户掩码区域动态分配令牌,高细节区保留更多令牌
- 在小、中、大掩码比下分别提速3.13倍、2.59倍、1.67倍
- 适合需要快速生成的生产级图像编辑场景
创意图像编辑工具如Photoshop的移除或生成填充功能是日常使用的核心,占Photoshop和Lightroom流量的很大比例。然而,当前生成式AI模型面临显著延迟问题,尤其在从卷积U-Net转向扩散Transformer(DiT)时更为突出。我们在数百个代表性图像编辑样本上评估发现,仅DiT模块就占总延迟的73%,即使将采样步数从50步压缩至8步后仍如此。为此,我们提出HiLo-Token,一种输入自适应的令牌压缩框架:对用户指定的编辑区域,通过膨胀掩码保留所有令牌以保持局部性和上下文相关性;在编辑区外,基于空间频率选择高频重要细节令牌,并用16×下采样的图像令牌表示低频成分,保留模糊但全局的结构。在生产级数据上的实验验证了该方法的有效性,在A100-80GB上对小、中、大掩码比(平均分别为6.38%、15.92%、35.36%)任务分别实现3.13倍、2.59倍、1.67倍的DiT加速,且生成质量无下降。
原文摘要 · Abstract (English)
Creative image editing tools, such as Photoshop's Remove or Generative Fill buttons, are central to everyday customer use and account for a major share of traffic in Photoshop and Lightroom. However, current generative AI models face significant latency challenges, which become even more pronounced when transitioning from convolution-based U-Nets to Diffusion Transformers (DiTs). In our evaluation on hundreds of representative image editing samples spanning a wide range of mask ratios, the DiT module alone accounts for an average of 73% of the total model latency, even after being distilled from 50 timesteps down to 8 timesteps. To tackle this challenge, we propose $\textbf{HiLo-Token}$, an input-adaptive token compression framework that allocates more token budget to high-frequency, rich-context regions while assigning fewer tokens to low-frequency areas. Specifically, for the editing region specified by the user mask, we retain all tokens within a dilated mask to preserve strong locality and contextual relevance. Outside the editing region, we introduce a simple yet effective high-frequency token selection strategy based on spatial frequency to capture important local details, while using tokens from a 16x downsampled image to represent low-frequency components and preserve the blurry but global structure. Extensive experiments on production-level evaluation data validate the effectiveness of the proposed method, achieving 3.13x, 2.59x, and 1.67x DiT speedups on A100-80GB for image editing tasks across small, medium, and large mask ratio categories with average ratios of 6.38%, 15.92%, and 35.36%, respectively, without any regression in generation quality.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。