arXiv:2507.01409cs.CV2025-07ICCV被引 5

让一张图的描述长短、详略可自由调节,像调音量一样简单。

CaptionSmiths: Flexibly Controlling Language Pattern in Image Captioning

  • 用数值量化描述长度、详尽度和词汇独特性,无需人工标注
  • 通过插值极端状态向量,实现语言风格平滑切换
  • 控制描述长度误差降低506%,适合多场景图文生成

图像描述模型若能灵活调整语言风格(如描述长度与详尽程度),将适用于更多应用场景。然而,尽管生成式视觉-语言模型进步显著,对生成描述属性的细粒度控制仍面临挑战:一是训练时未将属性作为条件输入,二是无法在不同语言模式间平滑过渡。为此,我们提出CaptionSmiths方法,使单一模型能处理多种语言模式。首先,我们无需人工标注,将每个描述的长度、详尽度及词汇独特性量化为连续标量;随后,通过插值两个端点向量(如极短与极长描述)实现条件控制。实验表明,该模型可平滑改变输出描述属性,且在词汇对齐上优于基线。例如,在控制描述长度方面,误差降低506%。代码将公开于https://github.com/omron-sinicx/captionsmiths。

原文摘要 · Abstract (English)

An image captioning model flexibly switching its language pattern, e.g., descriptiveness and length, should be useful since it can be applied to diverse applications. However, despite the dramatic improvement in generative vision-language models, fine-grained control over the properties of generated captions is not easy due to two reasons: (i) existing models are not given the properties as a condition during training and (ii) existing models cannot smoothly transition its language pattern from one state to the other. Given this challenge, we propose a new approach, CaptionSmiths, to acquire a single captioning model that can handle diverse language patterns. First, our approach quantifies three properties of each caption, length, descriptiveness, and uniqueness of a word, as continuous scalar values, without human annotation. Given the values, we represent the conditioning via interpolation between two endpoint vectors corresponding to the extreme states, e.g., one for a very short caption and one for a very long caption. Empirical results demonstrate that the resulting model can smoothly change the properties of the output captions and show higher lexical alignment than baselines. For instance, CaptionSmiths reduces the error in controlling caption length by 506\% despite better lexical alignment. Code will be available on https://github.com/omron-sinicx/captionsmiths.

图像描述可控生成语言风格

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。