让大模型在不更新参数的情况下，智能分配额外计算资源，提升推理精度。
HALO: Hybrid Adaptive Latent Reasoning for Language Models
- 分两阶段动态选择关键词元进行精细计算，避免无效处理。
- 在主流评测中超越固定1步和2步的改进方法，平均性能最优。
- 计算量比固定2步少得多，适合资源受限场景下的高效推理。
我们研究如何用少量自适应计算提升冻结的预训练语言模型性能。简单方法是在主干隐藏状态上增加固定次数的精炼步骤，但固定策略效率低：单步精炼可能太弱，而全局执行第二步全序列精炼会增加计算却未必提升迁移能力。本文提出HALO，一种混合自适应潜在精炼方法，结合粗粒度精炼与基于词元评分和单调停止机制的选择性二次精炼。在基于MMLU-Pro和GPQA-Diamond构建的主要公开基准测试中，HALO在论文面向方法中达到最佳平均表现，优于冻结主干、固定1步和固定2步方案。内部分析显示，HALO在词元准确率上接近固定2步水平，但平均应用精炼步数低于固定1步，远低于固定2步。结果表明，核心优势并非更多精炼，而是更优的计算分配：HALO在取得最强论文面向结果的同时，控制器计算开销也低于任一固定基线。
原文摘要 · Abstract (English)
We study how to improve a frozen pretrained language model with a small amount of adaptive extra computation. A simple approach is to add additional refinement steps on top of the backbone hidden states, but fixed extra refinement can be wasteful: a one-step refinement head may be too weak, while forcing a second full-sequence refinement step everywhere can increase compute without improving transfer. We introduce HALO, a hybrid adaptive latent-refinement method that combines a coarse refinement stage with selective second-stage latent refinement on a subset of tokens chosen by token scoring and monotonic token halting. On the main public benchmark comparison built from MMLU-Pro and GPQA-Diamond, HALO achieves the best overall average among the paper-facing methods, outperforming the frozen backbone, fixed-1, and fixed-2. Internal analysis further shows that HALO reaches nearly the same token-accuracy level as fixed-2 while using fewer average applied refine steps than fixed-1 and far fewer than fixed-2. These results suggest that the key advantage is not simply more refinement, but a better allocation of refinement: HALO achieves the strongest paper-facing result while also using less measured controller compute than either fixed baseline.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。