探究Transformer是否按任务难度自适应使用深度，发现微调后模型表现更明显。
Do Transformers Use their Depth Adaptively? Evidence from a Relational Reasoning Task
- 通过家庭关系多跳推理任务，用早期输出和因果修补分析各层作用
- 微调后模型在简单任务中用更少层数得合理答案，长链任务需更多层整合信息
- 非严格约束的微调方式让自适应效果更显著，适合研究模型内部机制
我们研究Transformer是否在不同难度的任务中自适应地使用其深度。基于家庭故事构建的受控多跳关系推理任务中，难度由需组合的关系跳跃数决定。通过早期读出（对数透镜）监测预测在各层的演变，以及通过因果修补分析任务相关信息在标记间的整合。对于预训练模型，发现有限的自适应证据：部分大模型在简单任务中可用较少层数得出合理答案，且随着链长增加，模型通常需要更多层数整合信息。对于在该任务上微调的模型，自适应深度使用的证据更清晰且一致，且在不严格保留通用语言建模能力的微调设置下效果更强。
原文摘要 · Abstract (English)
We investigate whether transformers use their depth adaptively across tasks of increasing difficulty. Using a controlled multi-hop relational reasoning task based on family stories, where difficulty is determined by the number of relationship hops that must be composed, we monitor (i) how predictions evolve across layers via early readouts (the logit lens) and (ii) how task-relevant information is integrated across tokens via causal patching. For pretrained models, we find some limited evidence for adaptive depth use: some larger models need fewer layers to arrive at plausible answers for easier tasks, and models generally use more layers to integrate information across tokens as chain length increases. For models finetuned on the task, we find clearer and more consistent evidence of adaptive depth use, with the effect being stronger for less constrained finetuning regimes that do not preserve general language modeling abilities.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。