用压缩文件结构直接训练Transformer，效率更高且精度不降。
Transformers from Compressed Representations
- 利用压缩文件的字节流结构设计令牌化方法
- 减少90%以上令牌数，同时保持高分类准确率
- 适合需要低资源推理的多媒体模型部署
压缩文件格式是高效数据存储与传输的核心，但其在表征学习中的潜力尚未被充分探索。我们提出TEMPEST（TransformErs from Compressed Representations），利用压缩文件固有的字节流结构，设计出一种有效的分词与编码策略。通过这种紧凑编码，标准Transformer可直接从压缩数据流中学习语义表征，无需原始字节级处理或完整媒体解码。该方法显著减少了语义分类所需的令牌数量，从而降低计算复杂度与内存占用。在多种数据集、编码方案和模态上的大量实验表明，TEMPEST在保持与当前最先进方法相当的准确率的同时，实现了内存与计算效率的显著提升。
原文摘要 · Abstract (English)
Compressed file formats are the corner stone of efficient data storage and transmission, yet their potential for representation learning remains largely underexplored. We introduce TEMPEST (TransformErs froM comPressed rEpreSenTations), a method that exploits the inherent byte-stream structure of compressed files to design an effective tokenization and encoding strategy. By leveraging this compact encoding, a standard transformer can directly learn semantic representations from compressed data streams, bypassing the need for raw byte-level processing or full media decoding. Our proposal substantially reduces the number of tokens required for semantic classification, thereby lowering both computational complexity and memory usage. Through extensive experiments across diverse datasets, coding schemes, and modalities, we show that TEMPEST achieves accuracy competitive wit the state-of-the-art while delivering efficiency gains in memory and compute.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。