动态合并冗余令牌,让边缘设备高效运行大模型。
Adaptive Token Merging for Efficient Transformer Semantic Communication at the Edge
- 按层动态合并相似令牌,不需重训练即可压缩表示。
- 图像分类任务中减少30%计算量,通信成本低于20%。
- 适合资源受限的边缘智能场景,还提升隐私安全。
大规模Transformer在现代语义通信中至关重要,但其高计算与通信开销限制了在资源受限边缘设备上的部署。本文提出一种无需训练的自适应令牌合并框架,通过在运行时根据每层语义相似度阈值选择性合并冗余令牌,实现对Transformer表示的压缩。与以往固定比率缩减不同,该方法将合并策略与输入冗余直接关联,实现数据驱动的自适应,无需重训练即可在效率与任务相关性间取得平衡。我们将合并策略发现建模为多目标优化问题,并利用贝叶斯优化获得精度、推理成本与通信成本间的帕累托最优解。在ImageNet分类任务中,模型精度与原始Transformer相当,但每秒浮点运算减少30%,通信成本低于原系统的20%;在视觉问答任务中,性能接近完整LLaVA模型,计算量不足三分之一,带宽需求仅为十分之一。此外,该方法对不同信道条件具有鲁棒性,并显著降低模型逆向攻击的有效性,具备内在隐私保护优势。本框架为在资源受限边缘环境中部署强大Transformer模型提供了实用且通用的解决方案。
原文摘要 · Abstract (English)
Large-scale transformers are central to modern semantic communication, yet their high computational and communication costs hinder deployment on resource-constrained edge devices. This paper introduces a training-free framework for adaptive token merging, a novel mechanism that compresses transformer representations at runtime by selectively merging semantically redundant tokens under per-layer similarity thresholds. Unlike prior fixed-ratio reduction, our approach couples merging directly to input redundancy, enabling data-dependent adaptation that balances efficiency and task relevance without retraining. We cast the discovery of merging strategies as a multi-objective optimization problem and leverage Bayesian optimization to obtain Pareto-optimal trade-offs between accuracy, inference cost, and communication cost. On ImageNet classification, we match the accuracy of the unmodified transformer with 30\% fewer floating-point operations per second and under 20\% of the original communication cost, while for visual question answering our method achieves performance competitive with the full LLaVA model at less than one-third of the compute and one-tenth of the bandwidth. Finally, we show that our adaptive merging is robust across varying channel conditions and provides inherent privacy benefits, substantially degrading the efficacy of model inversion attacks. Our framework provides a practical and versatile solution for deploying powerful transformer models in resource-limited edge intelligence scenarios.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。