首次揭示Transformer模型在长文本处理中的性能瓶颈
Transformer Scalability Crisis: The First Comprehensive Empirical Analysis of Performance Walls in Modern Language Models
- 系统测试118个模型，发现序列长度超512后成功率骤降
- 2048词元时模型完全失败，压缩模型效率是大模型的52倍
- 挑战了模型可无限扩展的假设，适合部署决策者阅读
尽管Transformer架构在自然语言处理中取得显著成功，但其可扩展性限制仍缺乏系统的实证分析。本文首次对118个Transformer模型在七种不同架构类别下进行大规模评估，揭示了表现为硬性部署限制的根本性能墙。系统基准测试方法发现：88.1%的模型能处理至512词元，但在1024词元时下降至44.9%，2048词元时完全失败（0%）。通过从128到2048词元的加载时间、内存消耗和计算效率分析，我们证明压缩模型实现更高参数效率（649.2 tokens/sec/M），远优于大型生成模型（12.5 tokens/sec/M）。研究首次提供量化证据，表明理论O(n²)注意力复杂度转化为实际性能墙。本工作建立新的Transformer评估基准方法，为生产环境部署提供关键洞见。
原文摘要 · Abstract (English)
Despite the remarkable success of transformer architectures in natural language processing, their scalability limitations remain poorly understood through systematic empirical analysis. This paper presents the first comprehensive large-scale evaluation of 118 transformer models across seven distinct architectural categories, revealing fundamental performance walls that manifest as hard deployment constraints. Our systematic benchmarking methodology uncovers a critical scalability crisis: while 88.1% of models successfully process sequences up to 512 tokens, this drops dramatically to 44.9% at 1024 tokens, with complete failure (0%) at 2048 tokens. Through rigorous analysis of loading times, memory consumption, and computational efficiency across sequence lengths from 128 to 2048 tokens, we demonstrate that compressed models achieve superior parameter efficiency (649.2 tokens/sec/M parameters) compared to large generative models (12.5 tokens/sec/M). Our findings challenge prevailing scaling assumptions and provide the first quantitative evidence that the theoretical O(n2) attention complexity translates into measurable performance walls. This work establishes new benchmarking methodologies for transformer evaluation and provides critical insights for practical deployment decisions in production environments.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。