arXiv:2505.14513cs.LGcs.AI2025-05被引 1

用单个可学习变换器替代多层,大幅压缩模型同时保持性能。

Latent Flow Transformer

  • 用流匹配训练单一传输算子替代多个网络层
  • 压缩6层后生成质量优于跳过2层,KL散度低至0.407
  • 结合流行走算法可压缩12层,接近流模型生成效果

Transformer作为大语言模型的标准架构,通常包含数十到上百个离散层。尽管层数越多性能越好,但这种设计效率低下,尤其相较于扩散与基于流的图像生成模型所展现的连续层优势。本文提出潜空间流Transformer(LFT),将一组层替换为通过流匹配训练的单一学习型传输算子,在保持原有架构兼容性的同时实现显著压缩。此外,针对现有流方法在保持耦合性方面的不足,引入流行走(FW)算法。在Pythia-410M模型上,使用流匹配训练的LFT压缩了24层中的6层,其语言模型对数概率的KL散度为0.407,优于直接跳过2层的0.529,验证了该设计的可行性。采用FW训练时,LFT进一步将12层压缩为一层,KL散度降至0.736,优于跳过3层的0.932,显著缩小了自回归与流模型生成范式之间的差距。

原文摘要 · Abstract (English)

Transformers, the standard implementation for large language models (LLMs), typically consist of tens to hundreds of discrete layers. While more layers can lead to better performance, this approach has been challenged as far from efficient, especially given the superiority of continuous layers demonstrated by diffusion and flow-based models for image generation. We propose the Latent Flow Transformer (LFT), which replaces a block of layers with a single learned transport operator trained via flow matching, offering significant compression while maintaining compatibility with the original architecture. Additionally, we address the limitations of existing flow-based methods in \textit{preserving coupling} by introducing the Flow Walking (FW) algorithm. On the Pythia-410M model, LFT trained with flow matching compresses 6 of 24 layers and outperforms directly skipping 2 layers (KL Divergence of LM logits at 0.407 vs. 0.529), demonstrating the feasibility of this design. When trained with FW, LFT further distills 12 layers into one while reducing the KL to 0.736 surpassing that from skipping 3 layers (0.932), significantly narrowing the gap between autoregressive and flow-based generation paradigms.

Transformer流模型模型压缩生成模型

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。