让大模型推理更准：不用重训，只在推理时动态调整
TARo: Token-level Adaptive Routing for LLM Test-time Alignment
- 用可学习的分词路由机制，在推理时引导模型逻辑思考
- 数学推理提升22.4%，临床和指令遵循任务也显著改善
- 小模型到大模型通用，适合需要强推理但不想重训的场景
大型语言模型（LLMs）具备强大的推理能力，但通常需昂贵的后训练才能达到高性能。近期的测试时对齐方法提供了一种轻量级替代方案，但主要聚焦于偏好对齐而非推理对齐。为此，我们提出分词级自适应路由（TARo），在推理阶段完全无需微调即可引导冻结的LLM实现结构化推理。具体而言，我们先在分步数学推导轨迹上训练奖励模型以捕捉细粒度逻辑一致性信号，再引入一个可学习的分词级路由器，自动控制奖励模型对基础模型的引导。大量实验表明，TARo相比基线模型推理性能提升最高达+22.4%，相比现有分词级测试时对齐方法提升+8.4%；同时在分布外临床推理（MedXpertQA）和指令遵循（AlpacaEval）任务中表现优异。此外，TARo无需重新训练即可从小模型泛化到大模型，将测试时对齐从偏好优化扩展为鲁棒、跨领域的推理增强。
原文摘要 · Abstract (English)
Large language models (LLMs) exhibit strong reasoning capabilities but typically require expensive post-training to reach high performance. Recent test-time alignment methods offer a lightweight alternative, but have been explored mainly for preference alignment rather than reasoning. To bridge this gap, we propose, Token-level Adaptive Routing (TARo), which steers frozen LLMs toward structured reasoning entirely at inference time. Specifically, we first train reward models on step-wise mathematical traces to capture fine-grained logical consistency signals, then introduce a learnable token-level router that automatically controls the guidance of the reward model to the base model. Extensive experiments show that TARo significantly improves reasoning performance by up to +22.4% over base model and +8.4% over existing token-level test-time alignment methods, while also boosting out-of-distribution clinical reasoning (MedXpertQA) and instruction following (AlpacaEval). Furthermore, TARo also generalizes from small to large backbones without retraining, extending test-time alignment from preference optimization to robust, cross-domain reasoning.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。