通过测试时循环优化，让预训练模型在不训练的情况下持续精炼语义。
Inner Loop Inference for Pretrained Transformers: Unlocking Latent Capabilities Without Training
- 测试时重复应用模型中部分层，实现隐状态迭代优化。
- 在多个基准上提升准确率，且轨迹更稳定、语义更清晰。
- 适合希望零训练提升性能的模型使用者。
深度学习架构尤其是Transformer，通常被视为层的组合。实际上，这些层常由残差路径（复制输入）和变换块输出相加构成。因此，各层的输入可视为传播隐状态的迭代优化。现有研究认为内部表示在不同层间共享，使早期阶段即可解码词元。机制可解释性进一步推测某些层充当精炼层。本文提出测试时内循环推理：在预训练模型中反复应用选定的层范围，延长精炼过程。在多个基准测试中，该方法带来稳定但适度的准确率提升。对隐状态轨迹的分析显示其演化更稳定，语义持续优化。结果表明，仅通过简单的测试时循环即可在冻结预训练模型中扩展计算，获得额外精炼效果。
原文摘要 · Abstract (English)
Deep Learning architectures, and in particular Transformers, are conventionally viewed as a composition of layers. These layers are actually often obtained as the sum of two contributions: a residual path that copies the input and the output of a Transformer block. As a consequence, the inner representations (i.e. the input of these blocks) can be interpreted as iterative refinement of a propagated latent representation. Under this lens, many works suggest that the inner space is shared across layers, meaning that tokens can be decoded at early stages. Mechanistic interpretability even goes further by conjecturing that some layers act as refinement layers. Following this path, we propose inference-time inner looping, which prolongs refinement in pretrained off-the-shelf language models by repeatedly re-applying a selected block range. Across multiple benchmarks, inner looping yields modest but consistent accuracy improvements. Analyses of the resulting latent trajectories suggest more stable state evolution and continued semantic refinement. Overall, our results suggest that additional refinement can be obtained through simple test-time looping, extending computation in frozen pretrained models.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。