测试时动态优化视觉语言模型推理,提升准确率且不增加计算负担
Aligning Large Vision-Language Models at Test Time: A Trajectory-Guided Structured Sampling Approach

- 用轨迹学习构建推理记忆库,分解复杂问题为有序推理步骤
- 测试时通过马尔可夫链蒙特卡洛迭代优化推理路径,提升逻辑一致性
- 在多数据集上显著提效,适合需要高精度推理的复杂视觉任务
后训练强化学习算法常用于对齐大型视觉语言模型与人类意图及视觉推理任务需求。然而,现有基于RL的对齐方法通常资源消耗大,且训练目标与推理分布存在不匹配。为此,我们提出一种新颖的测试时对齐方法,利用轨迹引导的结构化采样实现动态推理优化,从而更好地对齐视觉定位并确保逻辑一致性。该方法首先通过轨迹学习算法构建推理记忆库,将复杂问题求解分解为预定义的有序推理模式序列;随后在推理时,先从记忆库中收集轨迹以建立全局结构化推理先验,再使用迭代马尔可夫链蒙特卡洛(MCMC)算法对推理轨迹进行局部多目标精炼。在多个多模态推理数据集上的实验表明,该方法显著提升准确率,且推理开销可控。结果表明,轨迹引导的测试时采样是传统后训练对齐的有效且可扩展的替代方案,尤其适用于复杂视觉推理任务。
原文摘要 · Abstract (English)
Post-training reinforcement learning (RL) algorithms are commonly used to align large vision-language models (LVLMs) with human intent and the requirements of visual reasoning tasks. However, existing RL-based alignment methods are often resource-intensive and encounter mismatches between training objectives and inference-time distributions. To bridge this gap, we propose a novel test-time alignment approach that leverages trajectory-guided structured sampling for dynamic inference-time refinement, achieving better alignment with visual grounding and ensuring logical consistency. Our approach begins with curating a reasoning memory bank via a trajectory learning algorithm, which decomposes complex question solving into ordered sequences of predefined reasoning patterns. It subsequently accomplishes inference-time alignment by first collecting trajectories from reasoning memory bank to establish a global structural reasoning prior, and then using an iterative Markov Chain Monte Carlo (MCMC) algorithm for localized multi-objective refinement of the reasoning trace. Experiments across multiple multimodal reasoning datasets demonstrate that our approach significantly improves accuracy without incurring prohibitive inference overhead. These results establish trajectory-guided test-time sampling as a scalable and effective alternative to traditional post-training alignment, particularly for complex visual reasoning tasks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。