arXiv:2605.07120cs.LGstat.ML2026-05

研究模型如何在符号名无关时仍能正确分类,揭示了泛化关键机制。

When Symbol Names Should Not Matter: A Logistic Theory of Fresh-Symbol Classification

  • 用核逻辑回归分析变压器在符号重命名下的分类行为
  • 发现训练数据中的符号冲突会引入有限样本扰动,影响分类边界
  • 提出碰撞图与正则化结合的通用判据,适用于提示策略优化

模板任务已成为检验变压器是否基于抽象符号而非具体词元名称进行推理的清晰测试平台。本文研究固定标签分类问题,其中训练与测试样本共享潜在模板但词汇不重叠。不同于下一词预测,模型无需生成未见符号,而是需学习对符号重命名不变的决策规则。我们分析了在变压器核范式下的正则化核逻辑分类,主要结果将学习到的预测器分解为理想模板级分类器和由训练数据中意外词元重叠引起的有限样本扰动。我们通过彩色碰撞图刻画这些重叠,并证明了新符号分类的高概率边缘传递保证。该视角将模板分析扩展至逻辑分类,精炼了标量多样性条件:词汇大小控制平均碰撞率,而碰撞几何结构决定理想分类边缘是否保留。更广泛地,同一扰动框架适用于增强抽象输入,给出一个通用的边距-碰撞准则,用于判断提示策略是否提升新符号泛化能力。合成模板实验验证了正则化、样本量和变压器核结构的预测作用。

原文摘要 · Abstract (English)

Template tasks have emerged as a clean testbed for asking whether transformers reason with abstract symbols rather than concrete token names. We study the fixed-label classification version of this problem, where train and test examples share latent templates but may use disjoint vocabularies. Unlike next-token prediction, the model need not emit unseen symbols; it must learn a decision rule invariant to symbol renaming. We analyze regularized kernel logistic classification in the transformer-kernel regime. Our main result decomposes the learned predictor into an ideal template-level classifier and a finite-sample perturbation caused by accidental token overlaps in the training data. We encode these overlaps by a colored collision graph and prove high-probability margin-transfer guarantees for fresh-symbol classification. This perspective extends template-based analyses to logistic classification and refines scalar diversity conditions: vocabulary size controls the average rate of collisions, but collision geometry controls whether the ideal classification margin is preserved. More broadly, the same perturbation framework applies to abstraction-augmented inputs, yielding a general margin-versus-collision criterion for identifying when prompting strategies improve fresh-symbol generalization. Synthetic template experiments illustrate the predicted roles of regularization, sample size, and transformer-kernel structure.

符号推理泛化能力分类理论提示工程

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。