让视觉大模型像人一样逐步筛选信息,精准定位关键图像区域。
UniDoc-RL: Coarse-to-Fine Visual RAG with Hierarchical Actions and Dense Rewards

- 用分层动作逐步从粗到细找图,主动裁剪重点区域。
- 在三个基准上比现有方法最高提升17.7%。
- 适合需要精细视觉推理的任务,如复杂问答与文档分析。
检索增强生成(RAG)通过引入外部视觉知识扩展了大型视觉语言模型(LVLM)的能力。然而,现有的视觉RAG系统通常依赖通用检索信号,忽视了复杂推理所需的细粒度视觉语义。为此,我们提出UniDoc-RL,一种统一的强化学习框架,其中LVLM代理协同完成检索、重排序、主动视觉感知和推理。UniDoc-RL将视觉信息获取建模为具有分层动作空间的序列决策问题,逐步从粗粒度文档检索细化到细粒度图像选择与主动区域裁剪,使模型能抑制无关内容并聚焦信息密集区域。为实现有效端到端训练,我们引入密集多奖励机制,为每个动作提供任务感知监督。基于组相对策略优化(GRPO),UniDoc-RL在无需独立价值网络的情况下对齐多目标行为。为支持此训练范式,我们构建了一个高质量推理轨迹数据集,包含细粒度动作标注。在三个基准上的实验表明,UniDoc-RL持续优于现有最先进基线,相比之前的基于RL的方法最高提升17.7%。
原文摘要 · Abstract (English)
Retrieval-Augmented Generation (RAG) extends Large Vision-Language Models (LVLMs) with external visual knowledge. However, existing visual RAG systems typically rely on generic retrieval signals that overlook the fine-grained visual semantics essential for complex reasoning. To address this limitation, we propose UniDoc-RL, a unified reinforcement learning framework in which an LVLM agent jointly performs retrieval, reranking, active visual perception, and reasoning. UniDoc-RL formulates visual information acquisition as a sequential decision-making problem with a hierarchical action space. Specifically, it progressively refines visual evidence from coarse-grained document retrieval to fine-grained image selection and active region cropping, allowing the model to suppress irrelevant content and attend to information-dense regions. For effective end-to-end training, we introduce a dense multi-reward scheme that provides task-aware supervision for each action. Based on Group Relative Policy Optimization (GRPO), UniDoc-RL aligns agent behavior with multiple objectives without relying on a separate value network. To support this training paradigm, we curate a comprehensive dataset of high-quality reasoning trajectories with fine-grained action annotations. Experiments on three benchmarks demonstrate that UniDoc-RL consistently surpasses state-of-the-art baselines, yielding up to 17.7% gains over prior RL-based methods.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。