小模型也能高效压缩句子，用单个3KB令牌还原完整语义。
Tiny Transformers Excel at Sentence Compression
- 用1-3层Transformer将句子压缩至3KB令牌
- 单个令牌可完整还原标准英文句子
- 为优化大模型提供新思路：从子词转向文本片段
令人惊讶的是，平均每个英文单词仅需5--6字节的ASCII表示，却需高达24千字节才能输入大型语言模型。我们发现每个标记嵌入中仍有大量信息冗余空间。实验表明，1--3层变压器可将标准英文句子编码并解码为仅3千字节的单一令牌。本研究暗示小型网络也能学习生成有效英文句子，并提示可通过将子词嵌入转向更大文本片段来优化大型语言模型。
原文摘要 · Abstract (English)
It is staggering that words of the English language, which are on average represented by 5--6 bytes of ASCII, require as much as 24 kilobytes when served to large language models. We show that there is room for more information in every token embedding. We demonstrate that 1--3-layer transformers are capable of encoding and subsequently decoding standard English sentences into as little as a single 3-kilobyte token. Our work implies that even small networks can learn to construct valid English sentences and suggests the possibility of optimising large language models by moving from sub-word token embeddings towards larger fragments of text.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。