arXiv:2411.02452cs.CVeess.IV2024-11被引 19

针对无线VQA传输瓶颈,提出目标导向语义通信框架,提升准确率并降低延迟。

Goal-Oriented Semantic Communication for Wireless Visual Question Answering

  • 根据问题目标提取关键语义,优先传输相关图像区域
  • 在高斯白噪声信道下准确率提升49%,瑞利信道下提升59%
  • 适合边缘计算场景下的实时视觉问答系统

人工智能与计算机视觉的快速发展推动了视觉问答(VQA)等计算密集型应用的发展,该任务融合视觉感知与自然语言处理生成答案。为突破本地算力限制,边缘计算被引入以提供额外计算能力,但由此带来的通信挑战——如带宽有限、信道噪声和多径效应——严重影响大分辨率图像传输,进而降低VQA性能与用户体验。为此,本文提出一种目标导向语义通信(GSC)框架,聚焦于提取并传输最相关的语义信息以提升回答准确性与通信效率。通过基于边界框(BBox)的图像语义提取与排序方法,根据问题目标优先传输关键区域;进一步结合场景图(SG)方法处理复杂关系问题。实验表明,在加性高斯白噪声(AWGN)信道下,准确率提升最高达49%;在瑞利信道下提升达59%;相比传统比特传输方式,总延迟降低最多65%。

原文摘要 · Abstract (English)

The rapid progress of artificial intelligence (AI) and computer vision (CV) has facilitated the development of computation-intensive applications like Visual Question Answering (VQA), which integrates visual perception and natural language processing to generate answers. To overcome the limitations of traditional VQA constrained by local computation resources, edge computing has been incorporated to provide extra computation capability at the edge side. Meanwhile, this brings new communication challenges between the local and edge, including limited bandwidth, channel noise, and multipath effects, which degrade VQA performance and user quality of experience (QoE), particularly during the transmission of large high-resolution images. To overcome these bottlenecks, we propose a goal-oriented semantic communication (GSC) framework that focuses on effectively extracting and transmitting semantic information most relevant to the VQA goals, improving the answering accuracy and enhancing the effectiveness and efficiency. The objective is to maximize the answering accuracy, and we propose a bounding box (BBox)-based image semantic extraction and ranking approach to prioritize the semantic information based on the goal of questions. We then extend it by incorporating a scene graphs (SG)-based approach to handle questions with complex relationships. Experimental results demonstrate that our GSC framework improves answering accuracy by up to 49% under AWGN channels and 59% under Rayleigh channels while reducing total latency by up to 65% compared to traditional bit-oriented transmission.

视觉问答语义通信边缘计算

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。