用大模型提升人脸视频语义通信效率与体验
Large Generative Model-assisted Talking-face Semantic Communication System
- 用生成式模型将人脸视频转为高密度文本传输
- 构建私有知识库消除语义歧义，提升重建质量
- 接收端还原音色一致的高清说话人脸视频
生成式人工智能的快速发展不断揭示语义通信（SemCom）的潜力。然而，现有说话人脸语义通信系统仍面临带宽利用率低、语义模糊和用户体验下降等挑战。本文提出一种面向说话人脸视频通信的大生成模型辅助语义通信系统（LGM-TSC）。首先，在发送端基于FunASR模型引入生成式语义提取器（GSE），将语义信息稀疏的说话人脸视频转换为高信息密度的文本；其次，构建基于大语言模型（LLM）的私有知识库（KB），实现语义消歧与修正，并采用知识库-语义信道联合编码方案；最后，在接收端提出生成式语义重构器（GSR），利用BERT-VITS2与SadTalker模型将文本还原为匹配用户音色的高质量说话人脸视频。仿真结果验证了所提LGM-TSC系统的可行性和有效性。
原文摘要 · Abstract (English)
The rapid development of generative Artificial Intelligence (AI) continually unveils the potential of Semantic Communication (SemCom). However, current talking-face SemCom systems still encounter challenges such as low bandwidth utilization, semantic ambiguity, and diminished Quality of Experience (QoE). This study introduces a Large Generative Model-assisted Talking-face Semantic Communication (LGM-TSC) System tailored for the talking-face video communication. Firstly, we introduce a Generative Semantic Extractor (GSE) at the transmitter based on the FunASR model to convert semantically sparse talking-face videos into texts with high information density. Secondly, we establish a private Knowledge Base (KB) based on the Large Language Model (LLM) for semantic disambiguation and correction, complemented by a joint knowledge base-semantic-channel coding scheme. Finally, at the receiver, we propose a Generative Semantic Reconstructor (GSR) that utilizes BERT-VITS2 and SadTalker models to transform text back into a high-QoE talking-face video matching the user's timbre. Simulation results demonstrate the feasibility and effectiveness of the proposed LGM-TSC system.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。