不需训练即可提升图像描述的细节与准确性
Patch Matters: Training-free Fine-grained Image Caption Enhancement via Local Perception
- 将图像划分为语义和空间块，增强局部感知能力
- 通过分层聚合生成更完整、无幻觉的长句描述
- 适用于开源与闭源模型，无需重新训练
高质量图像描述对文本到图像生成、文本到视频生成及图文检索等跨模态应用至关重要。现有多模态大模型常生成缺乏细粒度信息或存在幻觉的描述，问题在开源与闭源模型中均普遍存在。受特征整合理论启发，本文提出一种‘分而聚’策略：先将图像分割为语义与空间块以提取细粒度信息，增强模型局部感知；再通过分层聚合生成全面的全局描述。过程中引入语义级过滤机制，抑制幻觉与不一致。该无训练流程可适配LLaVA-1.5、LLaVA-1.6、Mini-Gemini等开源模型，以及Claude-3.5-Sonnet、GPT-4o、GLM-4V-Plus等闭源模型。大量实验表明，该方法显著提升描述的细节性与可靠性，推动跨模态描述生成发展。
原文摘要 · Abstract (English)
High-quality image captions play a crucial role in improving the performance of cross-modal applications such as text-to-image generation, text-to-video generation, and text-image retrieval. To generate long-form, high-quality captions, many recent studies have employed multimodal large language models (MLLMs). However, current MLLMs often produce captions that lack fine-grained details or suffer from hallucinations, a challenge that persists in both open-source and closed-source models. Inspired by Feature-Integration theory, which suggests that attention must focus on specific regions to integrate visual information effectively, we propose a \textbf{divide-then-aggregate} strategy. Our method first divides the image into semantic and spatial patches to extract fine-grained details, enhancing the model's local perception of the image. These local details are then hierarchically aggregated to generate a comprehensive global description. To address hallucinations and inconsistencies in the generated captions, we apply a semantic-level filtering process during hierarchical aggregation. This training-free pipeline can be applied to both open-source models (LLaVA-1.5, LLaVA-1.6, Mini-Gemini) and closed-source models (Claude-3.5-Sonnet, GPT-4o, GLM-4V-Plus). Extensive experiments demonstrate that our method generates more detailed, reliable captions, advancing multimodal description generation without requiring model retraining. The source code are available at https://github.com/GeWu-Lab/Patch-Matters
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。