Transformer通过补全矩阵块，自动发现通用迭代求解算法。
Linear Transformers Implicitly Discover Unified Numerical Algorithms
- 仅用输入输出对和损失函数训练，无显式算法指导
- 统一更新规则在三种计算场景下均实现二阶收敛
- 适合需要自适应求解的科学计算与数据补全任务
我们在数百万个掩码低秩矩阵补全任务上训练线性注意力Transformer：每个提示是带有缺失块的掩码低秩矩阵，缺失部分可能是（i）标量预测目标，或（ii）Nyström外推中的未见核片。模型仅接收输入-输出对与均方损失，未获正常方程、手工迭代规则或任务关联提示。令人惊讶的是，训练后代数展开揭示了三个不同计算范式（完全可见、秩受限更新、分布式计算）下的相同参数无关更新规则。我们证明该规则在全批问题上实现二阶收敛，降低分布式迭代复杂度，并在秩受限注意力下保持精度。因此，仅通过补块任务训练的Transformer，隐式发现了一个跨越预测、估计与Nyström外推的统一、资源自适应迭代求解器，凸显上下文学习的强大能力。
原文摘要 · Abstract (English)
We train a linear attention transformer on millions of masked-block matrix completion tasks: each prompt is masked low-rank matrix whose missing block may be (i) a scalar prediction target or (ii) an unseen kernel slice of Nyström extrapolation. The model sees only input-output pairs and a mean-squared loss; it is given no normal equations, no handcrafted iterations, and no hint that the tasks are related. Surprisingly, after training, algebraic unrolling reveals the same parameter-free update rule across three distinct computational regimes (full visibility, rank-limited updates, and distributed computation). We prove that this rule achieves second-order convergence on full-batch problems, cuts distributed iteration complexity, and remains accurate with rank-limited attention. Thus, a transformer trained solely to patch missing blocks implicitly discovers a unified, resource-adaptive iterative solver spanning prediction, estimation, and Nyström extrapolation, highlighting a powerful capability of in-context learning.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。