用改进的LLM提升跨模态查询理解,解决视觉冗余问题。
An Enhanced Large Language Model For Cross Modal Query Understanding System Using DL-KeyBERT Based CAZSSCL-MPGPT
- 结合E-YOLO与知识图谱生成对象骨架,增强视觉理解
- 在COCO 2017和vqav2-val上准确率达99.14%和98.43%
- 适合做图像描述生成、多模态搜索系统的开发者参考
大型语言模型(LLMs)是先进的深度学习模型,用于理解和生成人类语言,并与图像等数据处理模型协同实现跨模态理解。然而,现有方法常受回音室效应影响,冗余视觉模式降低模型泛化性与准确性。为此,本文提出基于DL-KeyBERT的CAZSSCL-MPGPT增强框架,用于跨模态查询理解。数据集包含预处理的图文对,图像经由Easom-You Only Look Once(E-YOLO)进行目标分割,生成对象骨架,并通过条件随机知识图谱(CRKG)构建知识图谱。从知识图谱、骨架及分割对象中提取特征,利用福萨优化算法(FOA)筛选最优特征。文本则通过深度学习版KeyBERT进行词嵌入。最终,跨模态系统使用CAZSSCL-MPGPT生成准确且上下文相关的图像描述。该模型在COCO 2017数据集上达到99.14187362%准确率,在vqav2-val数据集上达98.43224393%。
原文摘要 · Abstract (English)
Large Language Models (LLMs) are advanced deep-learning models designed to understand and generate human language. They work together with models that process data like images, enabling cross-modal understanding. However, existing approaches often suffer from the echo chamber effect, where redundant visual patterns reduce model generalization and accuracy. Thus, the proposed system considered this limitation and developed an enhanced LLM-based framework for cross-modal query understanding using DL-KeyBERT-based CAZSSCL-MPGPT. The collected dataset consists of pre-processed images and texts. The preprocessed images then undergo object segmentation using Easom-You Only Look Once (E-YOLO). The object skeleton is generated, along with the knowledge graph using a Conditional Random Knowledge Graph (CRKG) technique. Further, features are extracted from the knowledge graph, generated skeletons, and segmented objects. The optimal features are then selected using the Fossa Optimization Algorithm (FOA). Meanwhile, the text undergoes word embedding using DL-KeyBERT. Finally, the cross-modal query understanding system utilizes CAZSSCL-MPGPT to generate accurate and contextually relevant image descriptions as text. The proposed CAZSSCL-MPGPT achieved an accuracy of 99.14187362% in the COCO dataset 2017 and 98.43224393% in the vqav2-val dataset.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。