用智能流程自动修正语言与视觉对象的错配,提升检测准确率。
Re-Aligning Language to Visual Objects with an Agentic Workflow
- 设计智能流程,通过动态调整图像和文本提示来修正语言描述
- 在0.18M数据上训练,使检测性能提升约50%
- 适合需要高质量语言-视觉对齐数据的研究者
基于语言的物体检测(LOD)旨在将视觉对象与语言表达对齐。现有方法依赖大量成对数据并利用视觉语言模型(VLM)自动生成类人语言描述以扩展训练数据。然而我们发现,VLM幻觉会导致物体描述错误(如名称、颜色、形状),降低语言-视觉对齐质量。为此,提出由大语言模型控制的智能工作流Real-LOD,包含规划、工具使用和反思三个步骤:首先自动分析当前状态并制定行动策略;接着自适应调整图像与文本提示并送入VLM重新描述物体;最后用另一大模型评估优化后的描述提供反馈。该过程循环进行,逐步提升语言描述准确性,实现语言与视觉对象的再对齐。构建了一个仅含0.18M图像的标注数据集,并在此基础上训练主流LOD模型,在标准基准上性能相比现有方法提升约50%。该工作证明,通过自动化语言-视觉精修,在扩大数据量的同时可有效保障数据质量,从对齐角度显著提升检测性能。
原文摘要 · Abstract (English)
Language-based object detection (LOD) aims to align visual objects with language expressions. A large amount of paired data is utilized to improve LOD model generalizations. During the training process, recent studies leverage vision-language models (VLMs) to automatically generate human-like expressions for visual objects, facilitating training data scaling up. In this process, we observe that VLM hallucinations bring inaccurate object descriptions (e.g., object name, color, and shape) to deteriorate VL alignment quality. To reduce VLM hallucinations, we propose an agentic workflow controlled by an LLM to re-align language to visual objects via adaptively adjusting image and text prompts. We name this workflow Real-LOD, which includes planning, tool use, and reflection steps. Given an image with detected objects and VLM raw language expressions, Real-LOD reasons its state automatically and arranges action based on our neural symbolic designs (i.e., planning). The action will adaptively adjust the image and text prompts and send them to VLMs for object re-description (i.e., tool use). Then, we use another LLM to analyze these refined expressions for feedback (i.e., reflection). These steps are conducted in a cyclic form to gradually improve language descriptions for re-aligning to visual objects. We construct a dataset that contains a tiny amount of 0.18M images with re-aligned language expression and train a prevalent LOD model to surpass existing LOD methods by around 50% on the standard benchmarks. Our Real-LOD workflow, with automatic VL refinement, reveals a potential to preserve data quality along with scaling up data quantity, which further improves LOD performance from a data-alignment perspective.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。