给Transformer注意力头加方向性路由，用极少参数提升模型表现。
Directional Routing in Transformers
- 用共享路由器为每个注意力头分配学习到的抑制方向，仅增加3.9%参数量。
- 关闭路由后事实回忆率趋近零，归纳准确率从93.4%暴跌至0.0%。
- 路由机制不可替代，适合研究模型可解释性与轻量化设计的人看。
我们提出方向性路由，一种轻量级机制，通过共享路由器为每个Transformer注意力头赋予可学习的抑制方向，参数增加仅3.9%。在单次训练中对比一个433M参数模型与其相同基线模型，并通过机械可解释性追踪其计算电路。路由成为模型主要计算路径。关闭路由后，所有8个测试提示的事实回忆率降至接近零，归纳准确率从93.4%降至0.0%。移除单个注意力头影响微乎其微：主驱动头被移除反而提高目标概率，归纳头即使缺少最强成员仍保持98.6%准确率。协调机制不可替代，但其所协调的组件可替换。模型还自发组织为两种模式：早期层为领域自适应路由，晚期层为固定句法剪枝，其中变化最小的层最关键（禁用后困惑度+42.6）。路由使困惑度相对基线降低31-56%，尽管下游多选基准尚未体现这些提升。
原文摘要 · Abstract (English)
We introduce directional routing, a lightweight mechanism that gives each transformer attention head learned suppression directions controlled by a shared router, at 3.9% parameter cost. We train a 433M-parameter model alongside an identical baseline in a single run, then trace the resulting circuits through mechanistic interpretability. Routing becomes the model's dominant computational pathway. Disabling it collapses factual recall to near-zero probability across all 8 test prompts and drops induction accuracy from 93.4% to 0.0%. Knocking out individual attention heads has negligible effect: the primary mover head's removal actually increases target probability, and induction heads retain 98.6% accuracy without their strongest member. The coordination mechanism is irreplaceable; the components it coordinates are not. The model also self-organizes, without explicit pressure, into two regimes: domain-adaptive routing in early layers and fixed syntactic pruning in late layers, where the least-varying layer is the most critical (+42.6 PPL when disabled). Routing reduces perplexity 31-56% relative to the baseline, though downstream multiple-choice benchmarks do not yet reflect these gains.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。