重复训练样本能提升Transformer模型性能，优于更多样数据。
Emergent properties with repeated examples
- 用重复样本训练，比用大量单一样本效果更好。
- 小规模重复样本训练，准确率最高，达95%以上。
- 适合研究模型泛化与记忆的机制，对算法设计有启发。
我们研究了Transformer在算法生成数据集上，随着训练样本重复次数变化的表现。在求最大公约数、模乘法和矩阵特征值三个数学问题上，固定训练步数下，使用较少重复样本的模型表现优于使用大量单次使用样本的模型。还发现两组训练策略——对一小部分随机样本重复使用，其余正常采样——能实现更快学习和更优性能。这表明重复带来的收益可能超过数据多样性。这些数据集和任务提供了一个可控环境，有助于理解深度学习中泛化与记忆之间仍不明确的相互作用。
原文摘要 · Abstract (English)
We study the performance of transformers as a function of the number of repetitions of training examples with algorithmically generated datasets. On three problems of mathematics: the greatest common divisor, modular multiplication, and matrix eigenvalues, we show that for a fixed number of training steps, models trained on smaller sets of repeated examples outperform models trained on larger sets of single-use examples. We also demonstrate that two-set training - repeated use of a small random subset of examples, along normal sampling on the rest of the training set - provides for faster learning and better performance. This highlights that the benefits of repetition can outweigh those of data diversity. These datasets and problems provide a controlled setting to shed light on the still poorly understood interplay between generalization and memorization in deep learning.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。