用视觉大模型动态生成增强图像,提升模型在复杂场景下的感知能力。
VACoT: Rethinking Visual Data Augmentation with VLMs
- 推理时动态调用图像增强,不依赖额外训练。
- 在13个基准上显著提升对异常和对抗样本的鲁棒性。
- 适合需要强泛化能力的视觉感知任务,如OCR防御。
尽管视觉数据增强仍是训练鲁棒视觉模型的核心手段,但在视觉语言模型(VLMs)中却关注不足,后者主要依赖大规模真实数据或合成多样性。这导致其在常规感知任务上表现不如传统模型。由于预训练与微调成本高昂,继续在增强数据上训练收益递减。本文提出视觉增强链式思考(VACoT),在模型推理阶段动态调用图像增强。通过引入去噪等后处理变换,VACoT显著提升对挑战性及分布外输入的鲁棒性,尤其在与OCR相关的对抗场景中。不同于仅限局部裁剪的先前方法,VACoT整合一套结构化通用视觉增强策略,在扩大查询图像视角的同时降低训练复杂度与计算开销,并采用条件奖励机制,鼓励必要增强并惩罚冗余响应,确保感知任务中推理简洁高效。我们在13个感知基准上广泛验证了VACoT的优越性,并引入AdvOCR以凸显后处理增强在对抗场景中的泛化优势。
原文摘要 · Abstract (English)
While visual data augmentation remains a cornerstone for training robust vision models, it has received limited attention in visual language models (VLMs), which predominantly rely on large-scale real data acquisition or synthetic diversity. Consequently, they may struggle with basic perception tasks that conventional models handle reliably. Given the substantial cost of pre-training and fine-tuning VLMs, continue training on augmented data yields limited and diminishing returns. In this paper, we present Visual Augmentation Chain-of-Thought (VACoT), a framework that dynamically invokes image augmentations during model inference. By incorporating post-hoc transformations such as denoising, VACoT substantially improves robustness on challenging and out-of-distribution inputs, especially in OCR-related adversarial scenarios. Distinct from prior approaches limited to local cropping, VACoT integrates a structured collection of general visual augmentations, broadening the query image views while reducing training complexity and computational overhead with efficient agentic reinforcement learning. We propose a conditional reward scheme that encourages necessary augmentation while penalizing verbose responses, ensuring concise and effective reasoning in perception tasks. We demonstrate the superiority of VACoT with extensive experiments on 13 perception benchmarks and further introduce AdvOCR to highlight the generalization benefits of post-hoc visual augmentations in adversarial scenarios.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。