自改进框架自动生成高精度视觉问答标注数据
AutoVQA-G: Self-Improving Agentic Framework for Automated Visual Question Answering and Grounding Annotation
- 用思维链推理实现细粒度视觉验证，提升数据一致性
- 通过记忆增强提示优化，逐步改进生成提示质量
- 适合需要高质量训练数据的视觉语言模型研究者
手动标注高质量的带定位信息的视觉问答（VQA-G）数据集对推动视觉语言模型（VLMs）发展至关重要，但难以规模化。现有自动化方法常因模型幻觉导致数据保真度不一致，且验证机制依赖简单启发式规则而脆弱。为此，我们提出AutoVQA-G，一种自改进的智能体框架，用于自动化VQA-G标注。该框架采用迭代优化循环：一致性评估模块利用思维链（CoT）推理进行细粒度视觉验证；基于反馈，记忆增强的提示优化智能体分析失败样本的批评，逐步优化生成提示。实验表明，AutoVQA-G生成的VQA-G数据集在视觉定位准确性上优于主流多模态大模型，为构建高保真数据以支持更鲁棒的VLM训练与评估提供了有效路径。代码开源：https://github.com/rohnson1999/AutoVQA-G
原文摘要 · Abstract (English)
Manual annotation of high-quality visual question answering with grounding (VQA-G) datasets, which pair visual questions with evidential grounding, is crucial for advancing vision-language models (VLMs), but remains unscalable. Existing automated methods are often hindered by two key issues: (1) inconsistent data fidelity due to model hallucinations; (2) brittle verification mechanisms based on simple heuristics. To address these limitations, we introduce AutoVQA-G, a self-improving agentic framework for automated VQA-G annotation. AutoVQA-G employs an iterative refinement loop where a Consistency Evaluation module uses Chain-of-Thought (CoT) reasoning for fine-grained visual verification. Based on this feedback, a memory-augmented Prompt Optimization agent analyzes critiques from failed samples to progressively refine generation prompts. Our experiments show that AutoVQA-G generates VQA-G datasets with superior visual grounding accuracy compared to leading multimodal LLMs, offering a promising approach for creating high-fidelity data to facilitate more robust VLM training and evaluation. Code: https://github.com/rohnson1999/AutoVQA-G
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。