让视觉语言模型生成连续输出，提升定位与控制精度
DRIFT: A Residual Flow Adapter for Decoding Continuous Outputs in Vision-Language Models
- 用粗略预测+流匹配精修的残差方式，简化连续输出建模
- 在视觉定位和机器人控制任务中超越多种回归与生成方法
- 适用于多种主流视觉语言模型，适配性强
许多现代视觉语言模型基于离散标记的自回归解码。尽管文本输出接口支持大规模预训练并在多样任务上实现强零样本泛化，但在需要精确连续输出的任务（如事件时间边界定位或机器人控制动作生成）中表现不佳。为此，我们提出DRIFT，一种将预训练视觉语言模型适配到连续解码任务的通用框架。DRIFT结合一个提供目标输出粗略估计的基线预测器，以及基于流匹配的生成精修模块，迭代优化预测结果。该残差形式将生成建模问题从学习全局输出分布转变为围绕强先验建模局部残差分布，显著降低优化难度。我们在感知与规划任务（包括视觉定位和机器人控制）上评估DRIFT。在多种任务与架构（涵盖多模态大模型、视觉语言代理和加权注意力模型）中，DRIFT始终优于一组强大的回归与生成基线方法。
原文摘要 · Abstract (English)
Many modern vision-language models (VLMs) build on autoregressive decoding of discrete tokens. While text-based output interfaces enable scalable pretraining and strong zero-shot generalization across diverse tasks, they are poorly suited for problems that require precise continuous outputs, such as localizing temporal boundaries of events or generating robotic control actions. To address this challenge, we propose DRIFT, a general framework for adapting pretrained VLMs to continuous decoding tasks. DRIFT combines a base predictor, which provides a coarse estimate of the target output, with a generative refinement module based on flow matching that iteratively improves the prediction. This residual formulation transforms the generative modeling problem from learning a global output distribution to modeling a localized residual distribution around a strong prior, substantially simplifying optimization. We evaluate DRIFT on both perception and planning tasks, including visual grounding and robotic control. Across multiple tasks and architectures spanning MLLMs, VLAs, and WAMs, DRIFT consistently outperforms a strong set of regression- and generative-based solutions.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。