提出简易基准测试，揭示Transformer如何通过相关性实现长距离检索。
Gaussian Match-and-Copy: A Minimalist Benchmark for Studying Transformer Induction
- 用纯二阶相关信号构造极简检索任务，分离记忆与检索机制。
- 实验表明模型能发展出类实际的匹配复制能力，且不同架构表现差异明显。
- 发现梯度下降隐式导向最大间隔分离，实现硬匹配选择，适合研究模型归纳偏置。
匹配复制是大语言模型推理时从上下文中检索匹配词元并复制其后续词元的核心机制。然而，在自然数据上理解该行为的涌现十分困难，因为检索与记忆相互纠缠。为解耦二者，我们引入高斯匹配复制（Gaussian Match-and-Copy, GMC），一种通过纯二阶相关信号隔离长距离检索的极简基准。数值研究表明，该任务保留了Transformer在实践中发展匹配复制电路的关键定性特征，并可区分不同架构的检索能力。我们还在简化注意力设置下分析优化动态：尽管回归目标允许多种解，包括非检索方案，但梯度下降在特定条件下会引导参数发散而方向趋于最大间隔分离器，从而实现硬匹配选择。我们证明了在达到零经验损失的轨迹下，该最大间隔对齐成立。
原文摘要 · Abstract (English)
Match-and-copy is a core retrieval primitive used at inference time by large language models to retrieve a matching token from the context then copy its successor. Yet, understanding how this behavior emerges on natural data is challenging because retrieval and memorization are entangled. To disentangle the two, we introduce Gaussian Match-and-Copy (GMC), a minimalist benchmark that isolates long-range retrieval through pure second-order correlation signals. Numerical investigations show that this task retains key qualitative aspects of how Transformers develop match-and-copy circuits in practice, and separates architectures by their retrieval capabilities. We also analyze the optimization dynamics in a simplified attention setting. Although many solutions are a priori possible under a regression objective, including ones that do not implement retrieval, we identify an implicit-bias regime in which gradient descent drives the parameters to diverge while their direction aligns with the max-margin separator, yielding hard match selection. We prove this max-margin alignment for GD trajectories that reach vanishing empirical loss under explicit technical conditions.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。