用句子嵌入压缩文本，让大模型生成更快更省算力
Text Compression for Efficient Language Generation
- 将文本压缩为句子嵌入，通过注意力机制逐句生成
- 算力效率提升一个量级，运行速度加快三倍
- 适合追求高效推理的部署场景，尤其小规模模型
我们挑战了大语言模型必须依赖子词标记进行高质量生成的普遍认知。为此，提出层级Transformer语言模型GPTHF，通过将文本压缩为句子嵌入并使用句子注意力机制实现文本生成。GPTHF保留GPT架构，仅通过动态稀疏注意力掩码修改标记间交互。实验表明，在小规模模型中，GPTHF相比同尺寸GPT模型，算力消耗降低一个量级，运行速度提升三倍。这一成果得益于独特的生成方法：缓存并复用句子嵌入，使输入的大部分内容可跳过网络中大段计算层。
原文摘要 · Abstract (English)
We challenge the prevailing assumption that LLMs must rely fully on sub-word tokens for high-quality text generation. To this end, we propose the "Generative Pretrained Thoughtformer" (GPTHF), a hierarchical transformer language model capable of text generation by compressing text into sentence embeddings and employing a sentence attention mechanism. GPTHF retains GPT's architecture, modifying only token interactions via dynamic sparse attention masks. Our experiments show that GPTHF achieves an up to an order of magnitude improvement in FLOPs efficiency and a threefold increase in runtime speed compared to equally-sized GPT models in the low-size regime. This is achieved through a unique generation method that caches and reuses sentence embeddings, allowing significant portions of the input to bypass large parts of the network.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。