解释了Transformer为何能从小样本推广到大样本，适用于点云和图数据。
From Small to Large: Generalization Bounds for Transformers on Variable-Size Inputs
- 基于连续域建模，分析离散采样下Transformer的误差边界。
- 误差受采样密度与数据流形内在维度决定，与输入长度无关。
- 理论在不同规模点云和图上验证，适用于几何数据建模场景。
Transformer展现出显著的‘尺寸泛化’能力，能从较短序列外推到更长序列。这种现象已在点云、图结构和自然语言等多类任务中被观察到。尽管实证成功，但其理论机制仍不明确。本文针对几何数据（如从流形采样的点云、从图论采样的图）建立理论框架，分析离散样本与连续域间输出误差的界。核心结论为：当位置编码稳定时，该误差由采样密度与数据流形的内在维度决定。在多种规模的点云与图数据上的实验验证了该边界的紧致性。
原文摘要 · Abstract (English)
Transformers exhibit a notable property of \emph{size generalization}, demonstrating an ability to extrapolate from smaller token sets to significantly longer ones. This behavior has been documented across diverse applications, including point clouds, graphs, and natural language. Despite its empirical success, this capability still lacks some rigorous theoretical characterizations. In this paper, we develop a theoretical framework to analyze this phenomenon for geometric data, which we represent as discrete samples from a continuous source (e.g., point clouds from manifolds, graphs from graphons). Our core contribution is a bound on the error between the Transformer's output for a discrete sample and its continuous-domain equivalent. We prove that for Transformers with stable positional encodings, this bound is determined by the sampling density and the intrinsic dimensionality of the data manifold. Experiments on graphs and point clouds of various sizes confirm the tightness of our theoretical bound.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。