用小模型聚合多个答案，提升大模型推理准确率
Learning to Reason Across Parallel Samples for LLM Reasoning
- 训练小模型并行分析多个生成答案，统一输出最终结果
- 在MATH数据集上比简单投票高8%的通过率（pass@5）
- 可兼容顶级闭源模型输出，适合部署于资源受限场景
增大测试时计算量能显著提升大语言模型（LLM）性能。通过生成多个答案并以启发式方式聚合（如多数投票或使用验证器排序），可在数学类任务中获得稳定提升。本文提出一种新方法：训练一个紧凑的样本集聚合器（SSA），将多个样本拼接后输入，输出最终答案，并通过强化学习优化准确率。在五个推理数据集上的实验表明，该方法兼具有效性与高效性。值得注意的是，SSA在MATH数据集上相比朴素多数投票提升8% pass@5。此外，30亿参数的SSA超越了720亿参数的基于模型的重排序方法。分析还显示，SSA具备良好的泛化能力，适用于不同样本数量、基础模型家族与规模及任务。通过分离生成与聚合环节，本方法可高效处理来自顶级闭源模型的输出。
原文摘要 · Abstract (English)
Scaling test-time compute brings substantial performance gains for large language models (LLMs). By sampling multiple answers and heuristically aggregate their answers (e.g., either through majority voting or using verifiers to rank the answers), one can achieve consistent performance gains in math domains. In this paper, we propose a new way to leverage such multiple sample set. We train a compact LLM, called Sample Set Aggregator (SSA), that takes a concatenated sequence of multiple samples and output the final answer, optimizing it for the answer accuracy with reinforcement learning. Experiments on five reasoning datasets demonstrate both the efficacy and efficiency of SSA. Notably, SSA improves over naive majority voting by 8% pass@5 on MATH. Furthermore, our 3B SSA surpasses model-based re-ranking with a much larger 72B process reward model. Our analysis also shows promising generalization ability of SSA, across sample set sizes, base model families and scales, and tasks. By separating LLMs to generate answers and LLMs to analyze and aggregate sampled answers, our approach can work with the outputs from premier black box models easily and efficiently.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。