让大模型生成多个答案并排序，用新投票法提升推理准确率
Ranked Voting based Self-Consistency of Large Language Models
- 每轮推理输出多个答案并按质量排序
- 在六项任务上超越基线，最高提升4.2个百分点
- 适合需要高可靠推理的开放问答场景
多数投票被视作提升链式思维推理的有效方法，通过选择不同推理路径中自一致性最高的答案实现（Wang等，2023）。然而，现有链式思维方法通常每次只生成一个答案，忽略了其他潜在答案的可能性，导致这些备选答案在后续投票中被忽略。本文提出在每轮推理中生成排序后的答案，并对多轮不同响应中的排序答案进行排序投票，从而提升整体自一致性可靠性。具体采用三种排序投票方法：即时淘汰投票、博尔达计数投票和平均倒数排名投票。我们在六个数据集上进行了验证，包括三个多项选择题和三个开放式问答任务，使用了先进的开源与闭源大模型。大量实验结果表明，所提方法优于基线，在多个任务上取得显著性能提升。代码已公开于 https://github.com/szu-tera/RankedVotingSC。
原文摘要 · Abstract (English)
Majority voting is considered an effective method to enhance chain-of-thought reasoning, as it selects the answer with the highest "self-consistency" among different reasoning paths (Wang et al., 2023). However, previous chain-of-thought reasoning methods typically generate only a single answer in each trial, thereby ignoring the possibility of other potential answers. As a result, these alternative answers are often overlooked in subsequent voting processes. In this work, we propose to generate ranked answers in each reasoning process and conduct ranked voting among multiple ranked answers from different responses, thereby making the overall self-consistency more reliable. Specifically, we use three ranked voting methods: Instant-runoff voting, Borda count voting, and mean reciprocal rank voting. We validate our methods on six datasets, including three multiple-choice and three open-ended question-answering tasks, using both advanced open-source and closed-source large language models. Extensive experimental results indicate that our proposed method outperforms the baselines, showcasing the potential of leveraging the information of ranked answers and using ranked voting to improve reasoning performance. The code is available at https://github.com/szu-tera/RankedVotingSC.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。