研究Transformer在程序执行轨迹生成中的局限性
An empirical study on the limitation of Transformers in program trace generation
- 用多种改进方法训练小型Transformer模型
- 在分布内表现良好，但跨长度等条件泛化失败
- 揭示了模型对程序长度等因子的系统性弱点
我们研究了Transformer在程序轨迹生成（PTG）任务中的表现，该任务要求模型为合成程序生成逐步执行轨迹。与现有算法问题不同，PTG通过长轨迹外化推理过程，每一步操作均简单直接。我们训练了具有多种改进的小型Transformer模型，包括替代位置编码、Softmax替换、混合模型和短卷积结构。尽管这些模型在分布内任务上取得了良好准确率，但在面对程序长度、轨迹步数等变化因素时表现出系统性泛化失败，尽管部分设计显著提升了泛化能力。
原文摘要 · Abstract (English)
We study Transformers on the task \emph{program trace generation} (PTG), where models produce step-by-step execution traces for synthetic programs. Unlike existing algorithmic problems, PTG externalizes reasoning through long traces where each step is trivial. We train small Transformers with diverse modifications, including alternative position encodings, softmax replacements, hybrid model, and short convolutions. While these models achieve strong in-distribution accuracy, they exhibit systematic failures when generalizing to various factors (e.g., program length, trace steps), though some designs significantly improve generalization.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。