用深度学习生成语义准确的印地语图像描述,突破英语主导的局限。
Using Deep Learning to Generate Semantically Correct Hindi Captions
- 融合视觉特征与注意力机制,基于VGG16+双向LSTM生成描述
- BLEU-1达0.59,BLEU-4达0.19,实现流畅印地语描述生成
- 为非英语图像描述任务提供可复用的多模态模型范式
利用计算机视觉与自然语言处理能力实现自动化图像字幕生成具有重要意义。尽管英语领域已有大量研究,但对流行外语如印地语的关注仍不足。本研究针对全球第四大语言印地语,构建多模态模型生成图像描述。采用Flickr8k数据集,结合Google Cloud Translator生成印地语字幕。使用VGG16、ResNet50、Inception V3等预训练CNN提取局部与全局视觉特征,结合单向与双向文本编码,并引入注意力层生成时间步权重,融合为句级特征向量。通过BLEU分数评估,注意力机制加持的双向LSTM与VGG16组合在BLEU-1上达到0.59,在BLEU-4上达0.19,表明生成结果语义准确且连贯。实验验证了该模型在印地语图像字幕生成中的有效性,为后续研究提供了可行框架。
原文摘要 · Abstract (English)
Automated image captioning using the content from the image is very appealing when done by harnessing the capability of computer vision and natural language processing. Extensive research has been done in the field with a major focus on the English language which gives the scope for further developments in the same with consideration of popular foreign languages. This research utilizes distinct models for translating the image caption into Hindi, the fourth most popular language across the world. Exploring the multi-modal architectures this research comprises local visual features, global visual features, attention mechanisms, and pre-trained models. Using google cloud translator on the image dataset from Flickr8k, Hindi image descriptions have been generated. Pre-trained CNNs like VGG16, ResNet50, and Inception V3 helped in retrieving image characteristics, while the uni-directional and bi-directional techniques of text encoding are used for the text encoding process. An additional Attention layer helps to generate a weight vector and, by multiplying it, combine image characteristics from each time step into a sentence-level feature vector. Bilingual evaluation understudy scores are used to compare the research outcome. Many experiments that serve as a baseline are done for the comparative analysis of the research. An image with a score of BLEU-1 is considered sufficient, whereas one with a score of BLEU-4 is considered to have fluid image captioning. For both BLEU scores, the attention-based bidirectional LSTM with VGG16 produced the best results of 0.59 and 0.19 respectively. The experiments conclude that researchs ability to produce relevant, semantically accurate image captions in Hindi. The research accomplishes the goals and future research can be guided by this research model.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。