让模型训练对齐测试时计算资源，提升大模型推理性能
Compute Aligned Training: Optimizing for Test Time Inference
- 将推理策略视为对基础模型的运算操作，设计新损失函数
- 实验证明在多种测试策略下，性能显著优于传统训练方法
- 适合关注推理效率与模型表现提升的研究者和工程师
随着测试时计算资源的扩展，大型语言模型（LLM）的性能得到了显著提升。然而，标准的后训练范式（如监督微调SFT和强化学习RL）仅优化单个样本的概率似然，与依赖输出聚合或筛选的测试时策略存在目标错配。本文提出计算对齐训练（Compute Aligned Training），将训练目标与测试时策略对齐。通过将推理策略建模为对基础策略的操作，推导出在应用这些策略时能最大化性能的新损失函数，并在SFT和RL中实现对应形式。实验表明，该方法显著提升了测试时的可扩展性，优于标准训练。
原文摘要 · Abstract (English)
Scaling test-time compute has emerged as a powerful mechanism for enhancing Large Language Model (LLM) performance. However, standard post-training paradigms, Supervised Fine-Tuning (SFT) and Reinforcement Learning (RL), optimize the likelihood of individual samples under a base policy, creating a misalignment with test time procedures that rely on aggregated or filtered outputs. In this work, we propose Compute Aligned Training, which aligns training objectives with test-time strategies. By conceptualizing inference strategies as operators on the base policy, we derive new loss functions that maximize performance when said strategies are applied. We instantiate such loss functions for SFT and RL across common test time strategies. Finally, we provide empirical evidence that this training method substantially improves test time scaling over standard training.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。