arXiv:2412.11836cs.CV2024-12被引 9

统一生成事实与浪漫幽默风格的图像描述,解决重复和生僻词问题。

UnMA-CapSumT: Unified and Multi-Head Attention-driven Caption Summarization Transformer

  • 融合事实与双风格(浪漫/幽默)描述,用注意力机制统一生成。
  • 在Flickr8K和FlickrStyle10K上实现BLEU-4达0.423,优于基线模型。
  • 适合需要多风格、高质量图像描述的应用场景。

图像字幕生成近年来备受关注,现有方法多分别生成事实性或风格化字幕,导致同一图像产生多个字幕,且存在词汇外和重复问题。本文提出统一注意力与多头注意力驱动的字幕摘要变压器(UnMA-CapSumT),结合改进的自适应注意力事实字幕模型(MAA-FIC)与风格分解双向LSTM(SF-Bi-ALSTM)生成浪漫与幽默两种风格的字幕。所提总结器将两类描述融合,生成连贯丰富的风格化字幕。通过引入fastText词嵌入与注意力词嵌入(fTA-WE),并结合带覆盖机制的指针-生成网络,有效缓解词汇外与重复问题。在Flickr8K与FlickrStyle10K子集上的实验表明,该框架在自动评估指标上表现优异,消融实验验证了各模块的有效性。

原文摘要 · Abstract (English)

Image captioning is the generation of natural language descriptions of images which have increased immense popularity in the recent past. With this different deep-learning techniques are devised for the development of factual and stylized image captioning models. Previous models focused more on the generation of factual and stylized captions separately providing more than one caption for a single image. The descriptions generated from these suffer from out-of-vocabulary and repetition issues. To the best of our knowledge, no such work exists that provided a description that integrates different captioning methods to describe the contents of an image with factual and stylized (romantic and humorous) elements. To overcome these limitations, this paper presents a novel Unified Attention and Multi-Head Attention-driven Caption Summarization Transformer (UnMA-CapSumT) based Captioning Framework. It utilizes both factual captions and stylized captions generated by the Modified Adaptive Attention-based factual image captioning model (MAA-FIC) and Style Factored Bi-LSTM with attention (SF-Bi-ALSTM) driven stylized image captioning model respectively. SF-Bi-ALSTM-based stylized IC model generates two prominent styles of expression- {romance, and humor}. The proposed summarizer UnMHA-ST combines both factual and stylized descriptions of an input image to generate styled rich coherent summarized captions. The proposed UnMHA-ST transformer learns and summarizes different linguistic styles efficiently by incorporating proposed word embedding fastText with Attention Word Embedding (fTA-WE) and pointer-generator network with coverage mechanism concept to solve the out-of-vocabulary issues and repetition problem. Extensive experiments are conducted on Flickr8K and a subset of FlickrStyle10K with supporting ablation studies to prove the efficiency and efficacy of the proposed framework.

图像字幕风格生成注意力机制多风格

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。