研究位置编码对抽象推理任务的影响，发现2D编码在数据少时表现更优
The role of positional encodings in the ARC benchmark
- 用CodeT5+为案例，分析位置编码对推理能力的限制
- 2D位置编码在数据稀缺场景下性能最优，超越旋转编码
- 适合关注小样本抽象推理与模型架构优化的研究者
抽象与推理语料库（ARC）挑战人工智能系统在极少训练数据下进行抽象推理，这虽对人类直观易行，却对机器学习模型极具难度。以CodeT5+为例，我们揭示了位置编码的局限如何阻碍推理并影响性能。本研究进一步考察了不同Transformer架构中位置编码的作用，凸显其对不同规模与配置模型的关键影响。通过对比多种策略，发现2D位置编码和旋转位置嵌入（Rotary Position Embedding）均具竞争力，但2D编码在数据受限场景中表现更佳，表明其在处理ARC任务时更具优势。
原文摘要 · Abstract (English)
The Abstraction and Reasoning Corpus challenges AI systems to perform abstract reasoning with minimal training data, a task intuitive for humans but demanding for machine learning models. Using CodeT5+ as a case study, we demonstrate how limitations in positional encoding hinder reasoning and impact performance. This work further examines the role of positional encoding across transformer architectures, highlighting its critical influence on models of varying sizes and configurations. Comparing several strategies, we find that while 2D positional encoding and Rotary Position Embedding offer competitive performance, 2D encoding excels in data-constrained scenarios, emphasizing its effectiveness for ARC tasks
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。