用隐层验证器加速生成模型推理，省时省显存
Tiny Inference-Time Scaling with Latent Verifiers
- 直接在扩散模型中间特征上验证，无需转像素空间
- 候选生成效率提升63.3%，显存占用减少14.5%
- 适合资源受限场景下的高效生成优化
推理时扩展通过验证器评分并筛选生成结果来提升生成模型性能。常用方法是使用多模态大语言模型（MLLM）作为验证器，虽有效但引入显著推理开销。扩散模型在自编码器隐空间运行以降低计算量，但现有MLLM验证器仍需将候选解码至像素空间再重新编码为视觉嵌入，造成冗余操作。本文提出隐藏状态验证器（VHS），直接在扩散变换器（DiT）单步生成器的中间隐藏表示上进行分析，无需解码至像素空间，从而降低每候选验证成本，同时性能不降或优于传统方法。在仅少量候选（小推理预算）条件下，相比标准MLLM验证器，VHS使联合生成与验证时间减少63.3%，计算量（FLOPs）降低51%，显存占用减少14.5%，并在GenEval上实现+2.7%性能提升。
原文摘要 · Abstract (English)
Inference-time scaling has emerged as an effective way to improve generative models at test time by using a verifier to score and select candidate outputs. A common choice is to employ Multimodal Large Language Models (MLLMs) as verifiers, which can improve performance but introduce substantial inference-time cost. Indeed, diffusion pipelines operate in an autoencoder latent space to reduce computation, yet MLLM verifiers still require decoding candidates to pixel space and re-encoding them into the visual embedding space, leading to redundant and costly operations. In this work, we propose Verifier on Hidden States (VHS), a verifier that operates directly on intermediate hidden representations of Diffusion Transformer (DiT) single-step generators. VHS analyzes generator features without decoding to pixel space, thereby reducing the per-candidate verification cost while improving or matching the performance of MLLM-based competitors. We show that, under tiny inference budgets with only a small number of candidates per prompt, VHS enables more efficient inference-time scaling reducing joint generation-and-verification time by 63.3%, compute FLOPs by 51% and VRAM usage by 14.5% with respect to a standard MLLM verifier, achieving a +2.7% improvement on GenEval at the same inference-time budget.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。