arXiv:2503.12926eess.SPcs.CV2025-03中稿 · IEEE Transactions …被引 20

通过设备-边缘协同压缩视觉特征,显著降低传输与计算延迟。

Task-Oriented Feature Compression for Multimodal Understanding via Device-Edge Co-Inference

  • 用密度峰值聚类减少视觉特征数量,降低传输和计算负担。
  • 采用可学习的熵模型自适应编码,实现最高52%传输开销下降。
  • 适合资源受限设备上的实时多模态任务,如视觉问答。

随着大型多模态模型(LMMs)的发展,多模态理解应用日益增多。由于大多数推理请求来自计算能力有限的边缘设备,主流推理流程是将输入数据直接传至边缘服务器完成全部计算。然而,该方式因边缘设备上行带宽有限导致传输延迟高,且视觉标记数量庞大引发显著计算延迟,阻碍了对时延敏感的任务并降低用户体验。为此,本文提出一种面向任务的特征压缩(TOFC)方法,在设备-边缘协同推理框架中,通过聚类合并视觉特征,并利用可学习、可选择的熵模型进行特征编码。具体而言,基于K近邻的密度峰值聚类方法减少视觉特征数量,从而降低数据传输与计算复杂度;随后,采用带有超先验的可学习熵模型对合并后的特征进行编码与解码,进一步减少传输开销。为提升压缩效率,根据视觉特征特性自适应选择多个熵模型,实现更精确的概率分布估计。在七个视觉问答基准上的综合实验验证了所提TOFC方法的有效性。结果表明,相比神经压缩方法ELIC,TOFC在保持相同任务性能的前提下,实现了最高52%的数据传输开销降低和63%的系统延迟减少。

原文摘要 · Abstract (English)

With the rapid development of large multimodal models (LMMs), multimodal understanding applications are emerging. As most LMM inference requests originate from edge devices with limited computational capabilities, the predominant inference pipeline involves directly forwarding the input data to an edge server which handles all computations. However, this approach introduces high transmission latency due to limited uplink bandwidth of edge devices and significant computation latency caused by the prohibitive number of visual tokens, thus hindering delay-sensitive tasks and degrading user experience. To address this challenge, we propose a task-oriented feature compression (TOFC) method for multimodal understanding in a device-edge co-inference framework, where visual features are merged by clustering and encoded by a learnable and selective entropy model before feature projection. Specifically, we employ density peaks clustering based on K nearest neighbors to reduce the number of visual features, thereby minimizing both data transmission and computational complexity. Subsequently, a learnable entropy model with hyperprior is utilized to encode and decode merged features, further reducing transmission overhead. To enhance compression efficiency, multiple entropy models are adaptively selected based on the characteristics of the visual features, enabling a more accurate estimation of the probability distribution. Comprehensive experiments on seven visual question answering benchmarks validate the effectiveness of the proposed TOFC method. Results show that TOFC achieves up to 52% reduction in data transmission overhead and 63% reduction in system latency while maintaining identical task performance, compared with neural compression ELIC.

多模态特征压缩边缘计算

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。