通过动态分配视觉证据,让扩散模型生成更快更省资源。
DAVET: Denoising-Aware Visual Evidence Trajectory Allocation for Diffusion Vision-Language Models

- 根据去噪阶段变化,智能分配视觉信息使用量。
- 在多个基准上平均提速55%,性能损失仅1.86%。
- 无需训练,适用于主流扩散视觉语言模型。
扩散视觉语言模型(dVLMs)在迭代去噪过程中持续依赖视觉证据,导致视觉条件计算成本高。与自回归解码不同,扩散生成会反复回溯整个响应,且视觉需求随去噪步骤动态变化。分析表明,视觉证据需求具有强烈阶段依赖性,因此我们提出训练无关的去噪感知视觉证据轨迹分配方法(DAVET)。该方法基于阶段条件的证据轨迹,利用操作需求设定证据储备,并通过轨迹风险调节每一步的分配。通过单次视觉编码构建多层级证据视图,分离何时、何地需要多少证据。在两个代表性dVLM模型LLaDA-V和LaViDa上评估,覆盖多个视觉理解任务,DAVET实现平均1.55倍加速,相对性能下降仅为1.86%,证明去噪感知的视觉证据分配可显著降低计算开销并保持生成质量。
原文摘要 · Abstract (English)
Diffusion vision-language models (dVLMs) iteratively denoise masked responses while conditioning each denoising step on visual evidence, making visual conditioning a substantial recurring inference cost. Unlike autoregressive decoding, diffusion generation repeatedly revisits the entire response as uncertainty evolves. Our analysis reveals that visual evidence demand is strongly step-dependent, motivating adaptive allocation across denoising steps. Existing inference acceleration methods operate through decoding-side strategies or visual token compression via pruning and merging, but do not explicitly treat visual evidence as a resource whose demand evolves across the diffusion process. Therefore, we present Denoising-Aware Visual Evidence Trajectory Allocation (DAVET), a training-free framework that allocates visual evidence according to the evolving generation state. Starting from a phase-conditioned evidence trajectory, the proposed allocation policy uses operation demand to set an evidence reserve whose allocation at each denoising step is modulated by trajectory risk. DAVET realizes the resulting budgets through a hierarchy of evidence views constructed from a single visual encoding, separating when and how much evidence is needed from how the evidence views are constructed. Evaluated on two representative dVLMs, LLaDA-V and LaViDa, across multiple visual-understanding benchmarks, DAVET achieves an average speedup of 1.55$\times$ with an average relative performance drop of 1.86\%, showing that denoising-aware visual evidence allocation can reduce visual conditioning cost while largely preserving generation quality.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。