探究变压器模型在真实数据中的记忆能力，发现嵌入维度是关键决定因素。
Capacity Matters: a Proof-of-Concept for Transformer Memorization on Real-World Data
- 用医学知识图谱生成合成文本，测试模型记忆能力
- 嵌入大小直接影响学习速度与记忆容量，层数增益有限
- 软最大激活函数更稳定，复杂数据提升最终记忆效果
本文研究了模型架构与数据配置如何影响生成式Transformer在真实世界数据上的经验记忆容量。模型使用源自系统化医学命名法（SNOMED）知识图谱的合成文本数据集进行训练：三元组表示静态连接，序列模拟复杂关系模式。结果表明，嵌入大小是学习速度和容量的主要决定因素，额外层数带来的收益有限，甚至在简单数据集上可能损害性能。激活函数起关键作用，Softmax展现出更高的稳定性和记忆容量。此外，数据集复杂性增加似乎能提升最终记忆效果。这些发现加深了对Transformer记忆机制的理解，并为基于结构化真实数据的模型设计优化提供了框架。
原文摘要 · Abstract (English)
This paper studies how the model architecture and data configurations influence the empirical memorization capacity of generative transformers. The models are trained using synthetic text datasets derived from the Systematized Nomenclature of Medicine (SNOMED) knowledge graph: triplets, representing static connections, and sequences, simulating complex relation patterns. The results show that embedding size is the primary determinant of learning speed and capacity, while additional layers provide limited benefits and may hinder performance on simpler datasets. Activation functions play a crucial role, and Softmax demonstrates greater stability and capacity. Furthermore, increasing the complexity of the data set seems to improve the final memorization. These insights improve our understanding of transformer memory mechanisms and provide a framework for optimizing model design with structured real-world data.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。