让视觉模型自己生成语言工具链,实现更精细的视觉推理。
Lang2Act: Fine-Grained Visual Reasoning through Self-Emergent Linguistic Toolchains
- 用自涌现语言动作构建可复用工具箱,替代固定外部工具。
- 在多个视觉任务上提升性能超4%,有效保留图像信息。
- 适合需要精细视觉理解的智能系统开发者参考。
视觉检索增强生成(VRAG)通过引入外部视觉文档来增强视觉语言模型(VLMs)的能力以回答查询。现有框架通常依赖预定义的外部工具扩展感知能力,且将视觉感知与后续推理过程分离,这种解耦设计可能导致视觉信息丢失,尤其是在执行图像裁剪等操作时。本文提出Lang2Act,通过自涌现的语言工具链实现细粒度视觉感知与推理。不同于调用固定外部引擎,Lang2Act收集自涌现的动作作为语言工具,并利用它们增强VLM的视觉感知能力。为此,我们设计了两阶段强化学习训练框架:第一阶段优化VLM自探索高质量动作以构建可复用的语言工具箱;第二阶段进一步优化VLM以高效利用这些工具进行下游推理。实验结果表明,Lang2Act显著提升了VLM的视觉感知能力,在多个任务上性能提升超过4%。所有代码与数据可在https://github.com/NEUIR/Lang2Act获取。
原文摘要 · Abstract (English)
Visual Retrieval-Augmented Generation (VRAG) enhances Vision-Language Models (VLMs) by incorporating external visual documents to address a given query. Existing VRAG frameworks usually depend on rigid, pre-defined external tools to extend the perceptual capabilities of VLMs, typically by explicitly separating visual perception from subsequent reasoning processes. However, this decoupled design can lead to unnecessary loss of visual information, particularly when image-based operations such as cropping are applied. In this paper, we propose Lang2Act, which enables fine-grained visual perception and reasoning through self-emergent linguistic toolchains. Rather than invoking fixed external engines, Lang2Act collects self-emergent actions as linguistic tools and leverages them to enhance the visual perception capabilities of VLMs. To support this mechanism, we design a two-stage Reinforcement Learning (RL)-based training framework. Specifically, the first stage optimizes VLMs to self-explore high-quality actions for constructing a reusable linguistic toolbox, and the second stage further optimizes VLMs to exploit these linguistic tools for downstream reasoning effectively. Experimental results demonstrate the effectiveness of Lang2Act in substantially enhancing the visual perception capabilities of VLMs, achieving performance improvements of over 4%. All code and data are available at https://github.com/NEUIR/Lang2Act.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。