通过智能分配计算资源，让大模型推理更高效精准。
Scaling LLM Inference with Optimized Sample Compute Allocation
- 提出OSCA算法，自动优化不同采样配置的计算分配。
- 代码生成任务中，用128倍少的算力达到更优准确率。
- 适合需要高效推理的开发、研究及智能体应用。
采样是大语言模型推理中的一项基础操作。为在有限算力下高效扩展推理能力，关键在于找到样本计算预算的最优分配：使用哪些采样配置（模型、温度、语言等）？每个配置生成多少样本？我们将其建模为学习问题，提出OSCA算法，通过寻找最优的混合推理配置来优化样本计算分配。实验表明，在代码生成任务中，采用学习到的混合分配策略，仅需128倍少的算力即可获得优于最优单配置的准确率；在4个推理任务上，算力需求降低25倍。OSCA在多轮智能体工作流中同样有效，于SWE-Bench任务中实现更高准确率，且比默认配置节省3倍算力。代码与生成结果已开源（https://github.com/LeiLiLab/OSCA）。
原文摘要 · Abstract (English)
Sampling is a basic operation in many inference-time algorithms of large language models (LLMs). To scale up inference efficiently with a limited compute, it is crucial to find an optimal allocation for sample compute budgets: Which sampling configurations (model, temperature, language, etc.) do we use? How many samples do we generate in each configuration? We formulate these choices as a learning problem and propose OSCA, an algorithm that Optimizes Sample Compute Allocation by finding an optimal mix of different inference configurations. Our experiments show that with our learned mixed allocation, we can achieve accuracy better than the best single configuration with 128x less compute on code generation and 25x less compute on 4 reasoning tasks. OSCA is also shown to be effective in agentic workflows beyond single-turn tasks, achieving a better accuracy on SWE-Bench with 3x less compute than the default configuration. Our code and generations are released at https://github.com/LeiLiLab/OSCA.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。