通过对抗性扰动视觉输入,提升强化学习中视觉推理的探索能力。
Boosting RL-Based Visual Reasoning with Selective Adversarial Entropy Intervention
- 用采样响应熵设计对抗目标,扰动视觉输入以扩大探索空间。
- 在多个数据集上显著提升策略模型的推理表现,域内/域外均有效。
- 适合研究视觉语言模型强化学习与鲁棒性增强的学者参考。
近期,强化学习(RL)已成为提升视觉语言模型(VLMs)推理能力的常用方法。现有基于RL的微调方法中,熵干预能有效提升探索能力,从而改善策略性能。然而,多数研究仅在策略优化阶段控制特定标记的熵更新,忽略了在RL采样阶段进行熵干预对性能的潜在提升作用。本文提出选择性对抗熵干预(SaEI),通过从采样响应熵中生成标记选择性对抗目标,扭曲视觉输入以增强策略熵。具体地,我们提出熵引导的对抗采样(EgAS),将采样响应熵作为对抗目标;对应的对抗梯度可攻击视觉输入,生成对抗样本,使策略在采样时探索更广的答案空间。同时,提出标记选择性熵计算(TsEC),在不破坏VLM事实知识的前提下最大化对抗攻击效果。在域内与域外数据集上的大量实验表明,该方法通过熵干预显著提升策略探索能力,进而增强推理性能。代码将在论文接收后发布。
原文摘要 · Abstract (English)
Recently, reinforcement learning (RL) has become a common choice in enhancing the reasoning capabilities of vision-language models (VLMs). Considering existing RL-based finetuning methods, entropy intervention turns out to be an effective way to benefit exploratory ability, thereby improving policy performance. Notably, most existing studies intervene in entropy by simply controlling the update of specific tokens during policy optimization of RL. They ignore the entropy intervention during the RL sampling that can boost the performance of GRPO by improving the diversity of responses. In this paper, we propose Selective-adversarial Entropy Intervention, namely SaEI, which enhances policy entropy by distorting the visual input with the token-selective adversarial objective coming from the entropy of sampled responses. Specifically, we first propose entropy-guided adversarial sampling (EgAS) that formulates the entropy of sampled responses as an adversarial objective. Then, the corresponding adversarial gradient can be used to attack the visual input for producing adversarial samples, allowing the policy model to explore a larger answer space during RL sampling. Then, we propose token-selective entropy computation (TsEC) to maximize the effectiveness of adversarial attack in EgAS without distorting factual knowledge within VLMs. Extensive experiments on both in-domain and out-of-domain datasets show that our proposed method can greatly improve policy exploration via entropy intervention, to boost reasoning capabilities. Code will be released once the paper is accepted.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。