让视觉语言模型更准看懂图像,避免因误判导致错误操作。
CPPO: Contrastive Perception Policy Optimization for VLM Agents
- 用对比学习直接训练模型感知图像,无需额外标注或判断模型。
- 在干扰图像下识别关键感知词,只对这些词施加优化信号。
- 提升模型在需要精准视觉理解的任务表现,适合机器人等实际应用。
我们提出CPPO,一种用于微调视觉-语言模型(VLM)的对比感知策略优化方法。可靠的感知是开放环境中基于VLM的智能体推理与行动的核心要求:视觉定位错误会直接导致错误动作、幻觉工具调用和不安全决策。尽管强化学习(RL)显著提升了语言模型的推理能力,但将其推广到多模态智能体需同时改进感知与推理。以往工作主要依赖显式的感知奖励,常需额外大模型裁判、真实标注或强制分离感知与推理。CPPO通过自监督方式解决此问题,将强化学习目标扩展为对比感知损失(CPL),为视觉接地提供直接学习信号。该对比目标使模型更敏感于输入视觉信息。为有效应用此信号,CPPO利用扰动图像下输出分布熵的变化,识别感知相关词元,并在训练中选择性地对这些词元施加对比损失。实验表明,CPPO优于先前方法,且无需额外模型,训练更高效可扩展,生成的策略更适用于感知关键的智能体任务。
原文摘要 · Abstract (English)
We introduce CPPO, a Contrastive Perception Policy Optimization method for finetuning vision--language models (VLMs). Reliable perception is a core requirement for VLM-based agents that must reason and act in open-ended environments: faulty visual grounding cascades directly into faulty actions, hallucinated tool calls, and unsafe decisions. While reinforcement learning (RL) has significantly improved reasoning in language models, extending these advances to multimodal agents requires improving both perception and reasoning. Prior works address this challenge mainly through explicit perception rewards, which often require extra LLM judges, ground-truth annotations, or forced separation of perception from reasoning. CPPO addresses this limitation in a self-supervised manner by extending the RL objective with a Contrastive Perception Loss (CPL) that provides a direct learning signal for visual grounding. The contrastive objective encourages the model to become more sensitive to input visual information. To apply this signal effectively, CPPO identifies perception tokens using an entropy-shift mechanism in the model's output distributions under perturbed images and applies the contrastive loss selectively to those tokens during training. Experiments show that CPPO surpasses prior methods while avoiding extra models, making training more efficient and scalable, and yielding policies that are better suited to perception-critical agentic tasks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。