发现Transformer在方向性学习上存在固有偏差,即使无语义数据也难逆转训练。
Directional Optimization Asymmetry in Transformers: A Synthetic Stress Test
- 用可控熵的随机字符串构造正向/反向任务,隔离方向性影响
- GPT-2在反向任务中损失高出1.16 nats(K=5),远超MLP
- 该偏差源于架构本身,与语言统计无关,适合研究模型机制
Transformer理论上具备方向对称性,但实证中反复出现‘反转诅咒’。本文设计一个全合成、熵可控的基准测试,作为纯净环境下的方向性学习压力测试。通过调节分支因子K生成零条件熵的正向任务和具有解析熵下限的反向任务,发现即使从头训练的GPT-2模型在反向任务中仍表现出显著且可复现的方向优化差距(如K=5时达1.16 nats),远超同数据下训练的MLP。预训练初始化虽改变优化行为,但无法消除此差距;而LoRA在高熵反向映射上遭遇尖锐容量瓶颈。结果表明,因果Transformer训练存在一种最小化、无语义的方向摩擦,即便移除语言先验、词频与语料时间不对称后依然存在。该基准为剖析现代序列模型中的方向偏差提供了可控工具,并推动对反转为何始终更难的根本机制研究。
原文摘要 · Abstract (English)
Transformers are theoretically reversal-invariant: their function class does not prefer left-to-right over right-to-left mappings. Yet empirical studies on natural language repeatedly report a "reversal curse," and recent work on temporal asymmetry in LLMs suggests that real-world corpora carry their own arrow of time. This leaves an unresolved question: do directional failures stem from linguistic statistics, or from the architecture itself? We cut through this ambiguity with a fully synthetic, entropy-controlled benchmark designed as a clean-room stress test for directional learning. Using random string mappings with tunable branching factor K, we construct forward tasks with zero conditional entropy and inverse tasks with analytically determined entropy floors. Excess loss above these floors reveals that even scratch-trained GPT-2 models exhibit a strong, reproducible directional optimization gap (e.g., 1.16 nats at K=5), far larger than that of an MLP trained on the same data. Pre-trained initializations shift optimization behavior but do not eliminate this gap, while LoRA encounters a sharp capacity wall on high-entropy inverse mappings. Together, these results isolate a minimal, semantics-free signature of directional friction intrinsic to causal Transformer training-one that persists even when linguistic priors, token frequencies, and corpus-level temporal asymmetries are removed. Our benchmark provides a controlled instrument for dissecting directional biases in modern sequence models and motivates deeper mechanistic study of why inversion remains fundamentally harder for Transformers.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。