用强化学习优化模型推理速度，提升代码生成准确率。
Optimizing Language Models for Inference Time Objectives using Reinforcement Learning
- 用强化学习直接优化推理时的性能指标
- 代码生成任务中pass@k显著优于基线方法
- 适合关注推理效率与准确率的研究者
本文研究在模型训练阶段显式优化推理时间算法性能的可行性。通过在推理阶段使用$ k $个样本的通用目标，重点考察pass@$k$和多数投票两个典型应用。在推理数据集上训练语言模型时，展示了该方法带来的性能权衡。在代码生成任务中，该方法相比基线显著提升了pass@$k$指标，证明了其在提升模型整体效能方面的有效性。
原文摘要 · Abstract (English)
In this work, we investigate the merits of explicitly optimizing for inference time algorithmic performance during model training. We show how optimizing for inference time performance can improve overall model efficacy. We consider generic inference time objectives with $k$ samples, with a focus on pass@$k$ and majority voting as two main applications. With language model training on reasoning datasets, we showcase the performance trade-off enabled by training with such objectives. When training on code generation tasks, we show that the approach significantly improves pass@$k$ objectives compared to the baseline method.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。