用125M小模型实现高效图像描述，通过多轮细化提升准确性。
MM-SeR: Multimodal Self-Refinement for Lightweight Image Captioning
- 用125M参数小模型替代大模型，降低93倍计算开销。
- 通过粗细分层生成，使单句和细节描述均优于传统方法。
- 适合部署在资源受限的机器人、聊天系统等实时场景。
视频聊天机器人和导航机器人常依赖流式图像描述来理解视觉输入。现有方法通常采用大型多模态语言模型（MLLMs），但其高昂的计算成本限制了实际应用。为此，我们开发了一种轻量级描述模型。研究发现，将MLLM中的大规模语言组件替换为仅125M参数的小模型后，尽管规模缩小93倍，性能仍可媲美大模型，表明事实性图像描述并不需要大模型的复杂推理能力。然而，该轻量模型仍存在可靠性不足问题。受人类视觉过程启发——先感知全局粗略场景，再关注细节——我们提出一种多模态自精炼框架：利用前一轮粗略描述所提示的显著区域特征，生成更精确的细化描述。实验表明，本模型在单句与详细描述任务中均表现优越，并扩展至长序列视频问答任务。
原文摘要 · Abstract (English)
Systems such as video chatbots and navigation robots often depend on streaming image captioning to interpret visual inputs. Existing approaches typically employ large multimodal language models (MLLMs) for this purpose, but their substantial computational cost hinders practical application. This limitation motivates our development of a lightweight captioning model. Our investigation begins by replacing the large-scale language component in MLLMs with a compact 125M-parameter model. Surprisingly, this compact model, despite a 93x reduction in size, achieves comparable performance to MLLMs, suggesting that factual image captioning does not significantly require the complex reasoning abilities of LLMs. Despite this promising result, our lightweight model still lacks reliability. To address this, we draw inspiration from the human visual process: perceiving a global and coarse understanding of the scene before attending to finer details. Accordingly, we propose a multimodal self-refinement framework that guides the model to utilize features from salient regions, identified by referencing the previous coarse caption, and to produce a refined description. Experimental results demonstrate the superiority of our model in both single-sentence and detailed captioning, extending even to long-range video QA tasks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。