研究输入方式如何影响小模型少样本绑定能力，发现可读性和参数共享是关键。
Input Pathways Shape Few-Shot, Not Zero-Shot, Binding in Tiny Transformers: A Fully-Enumerable Study
- 在穷举数据集上测试微型Transformer，验证不同输入路径对组合绑定的影响。
- 少样本绑定效率由输入可读性与参数共享共同决定，符号化输入并非最优。
- 模型失败原因明确：符号路径在输出层丢失答案，索引路径错绑信息槽。
我们研究了信息以符号令牌、清晰的分因子‘理想’编码或纠缠感知向量等形式进入Transformer时，如何影响其是否能组合性地绑定信息。在约6-10K参数的Transformer上，基于穷举枚举的有限因子世界进行实验，使每项测量覆盖全部输入空间（零采样方差），且信息路径与任务匹配（精确贝叶斯上限为1.0）。报告四项发现：(1) 终点不变性：在未见绑定查询中，所有信息路径均未达到零样本组合成功，即使理论上限为1.0，说明失败源于归纳偏置而非信息缺失；(2) 少样本绑定的双因素解释：样本效率主要由输入路径的参数共享和可读性决定；通过维度匹配对照和可读性梯度实验，确认理想编码并非最高效可读路径；(3) 双分离现象：训练初期，分布式编码经历短暂高于随机的阶段（追踪编码格式），而少样本效率则与路径共享相关；(4) 失败机理：符号路径在输出层丢失答案，索引路径产生错误绑定（答案仍可解码，但干预显示输出追踪错误槽位），纠缠路径继承输入可读性。核心主张为双因素解释，终点与失败分析结果提供诊断约束。所有代码、配置与种子日志均已公开以实现完全复现。
原文摘要 · Abstract (English)
How does the way information reaches a transformer -- as symbolic tokens, a clean per-factor "oracle" code, or an entangled perceptual vector -- shape whether it binds that information compositionally? We study ~6-10K-parameter transformers on finite factored worlds enumerated exhaustively, so every measurement covers the whole input space (zero sampling variance) and the informative routes are information-matched (exact Bayes ceiling 1.0). We report four findings. (1) Endpoint invariance: on held-out binding queries no informative route reaches converged zero-shot composition -- each ends at or below chance despite a ceiling of 1.0, so within a bounded sweep the failure reflects inductive bias under a lookup-sufficient objective, not missing information. (2) A two-factor account of few-shot binding: sample efficiency is best explained by input-pathway parameter sharing and code readability; a dimension-matched control and a graded readability sweep isolate readability from input dimension, and the clean oracle is not the most sample-efficient readable route. (3) A double dissociation: early in training, distributed -- but not index-like -- codes pass through a transient above-chance phase (tracking code format), while few-shot efficiency tracks pathway sharing. (4) Failure anatomy: symbolic routes lose the answer at the readout; index routes mis-bind (the answer stays decodable, yet an input intervention shows the output tracks the wrong slot); entangled routes inherit their input's readability. The central claim is the two-factor account; the endpoint and anatomy results are diagnostic constraints. All code, manifests, and per-seed logs are released for exact reproduction.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。