用视觉变压器的令牌对齐技术,实现边缘设备与服务器间的高效语义通信。
Token-Oriented Semantic Communication with Pretrained Vision Transformers

- 通过令牌级相关性选择传输压缩后的图像隐向量,避免直接传令牌嵌入。
- 在ImageNet上比现有语义通信方案更优,兼顾传输率与识别准确率。
- 无需端到端训练,适合资源受限的边缘计算场景,可快速部署。
令牌通信在Transformer令牌粒度上实现了语义通信原理,为资源受限边缘系统中的客户端-服务器协同推理提供了新方向。然而,直接传输令牌嵌入面临通信开销大和跨模型嵌入空间互操作性差两大挑战。为此,本文提出一种面向令牌的语义通信框架:基于视觉变压器(ViT)块令牌与图像压缩(LIC)隐向量的一一空间对齐,使令牌级任务相关性直接决定传输哪些隐向量。该框架模块化设计,协调三个预训练组件——轻量级客户端ViT、学习型图像压缩(LIC)模型与大型服务器端ViT——无需端到端训练。关键机制包括:令牌对齐的LIC选择性传输任务相关隐向量;单次前向传播内层选择性注意力回溯估计令牌相关性;代理令牌替换通过优化单一可学习令牌适配冻结的服务器模型。ImageNet实验表明,该框架在速率-精度权衡上优于近期语义通信方案、手工编码器及任务无关的LIC模型。
原文摘要 · Abstract (English)
Token communications realize the semantic communication principle at the granularity of transformer tokens, providing a promising direction for client--server collaborative inference in resource-constrained edge systems. However, directly transmitting token embeddings presents two practical challenges: substantial communication cost and limited interoperability across model-specific token embedding spaces. To address these challenges, we propose a \emph{token-oriented} semantic communication framework. In this framework, token-level task relevance determines which compressed image latents are transmitted, enabling token-granular transmission without directly transmitting token embeddings. The framework is modular, coordinating three pretrained components---a lightweight client-side vision transformer (ViT), a learned image compression (LIC) model, and a large server-side ViT---without end-to-end training. The key enabler is the one-to-one spatial alignment between ViT patch tokens and the LIC latent vectors, which allows token-level task relevance to directly determine which latent vectors are transmitted. Building on this alignment, token-aligned LIC selectively transmits task-relevant latents, layer-selective attention rollout estimates token relevance from a selected range of attention layers in a single forward pass, and surrogate token substitution adapts the frozen server model by optimizing a single learnable token. Experiments on ImageNet show that the proposed framework achieves a more favorable rate--accuracy trade-off than recent semantic communication schemes, hand-crafted codecs, and task-agnostic LIC models.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。