实现多语言视频口型同步的异步并行框架,显著降低实时通信延迟。
Asynchronous Pipeline Parallelism for Real-Time Multilingual Lip Synchronization in Video Communication Systems
- 采用消息队列解耦的流水线并行架构,模块并发执行
- 端到端延迟降低3.1倍,提升处理速度与同步稳定性
- 适合物联网等资源受限场景,如远程医疗和多语言自助服务
本文提出一种用于实时视频会议系统中多语言口型同步的并行异步Transformer框架。该架构将翻译、语音处理和口型同步模块集成于流水线并行设计中,通过基于消息队列的解耦实现模块并发执行,相比串行方法端到端延迟降低最多3.1倍。为提升计算效率与吞吐量,各模块推理流程通过低级图编译、混合精度量化及硬件加速内核融合进行优化,在保持模型准确率和视觉质量的同时获得显著性能提升。此外,引入上下文自适应静音检测组件,按语义连贯边界分割输入语音流,增强跨语言翻译一致性与时序对齐效果。实验表明,所提并行架构在处理速度、同步稳定性和资源利用率方面均优于传统串行流水线。模块化、消息驱动的设计使其适用于资源受限的物联网通信场景,如远程医疗、多语言自助终端和远程协助系统。整体推动了下一代AIoT系统中低延迟、高效率多模态通信框架的发展。
原文摘要 · Abstract (English)
This paper introduces a parallel and asynchronous Transformer framework designed for efficient and accurate multilingual lip synchronization in real-time video conferencing systems. The proposed architecture integrates translation, speech processing, and lip-synchronization modules within a pipeline-parallel design that enables concurrent module execution through message-queue-based decoupling, reducing end-to-end latency by up to 3.1 times compared to sequential approaches. To enhance computational efficiency and throughput, the inference workflow of each module is optimized through low-level graph compilation, mixed-precision quantization, and hardware-accelerated kernel fusion. These optimizations provide substantial gains in efficiency while preserving model accuracy and visual quality. In addition, a context-adaptive silence-detection component segments the input speech stream at semantically coherent boundaries, improving translation consistency and temporal alignment across languages. Experimental results demonstrate that the proposed parallel architecture outperforms conventional sequential pipelines in processing speed, synchronization stability, and resource utilization. The modular, message-oriented design makes this work applicable to resource-constrained IoT communication scenarios including telemedicine, multilingual kiosks, and remote assistance systems. Overall, this work advances the development of low-latency, resource-efficient multimodal communication frameworks for next-generation AIoT systems.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。