通过聚类推理轨迹减少评分开销，提升大模型推理效率
VecCISC: Improving Confidence-Informed Self-Consistency with Reasoning Trace Clustering and Candidate Answer Selection
- 用语义相似性聚类过滤重复/错误推理路径
- 在5个领域数据集上降低47%的总调用量，精度不降反升
- 适合追求高效推理且重视准确率的研究者与应用方
自一致性推理中，加权多数投票（如基于置信度的自一致性，CISC）通过为每个候选答案分配置信度得分来提升准确率。但该方法需对每个推理路径调用一个评论模型生成置信度，带来显著计算开销。为此，本文提出轻量级自适应框架VecCISC，利用语义相似性识别并过滤语义等价、退化或幻觉的推理轨迹，减少需评估的候选答案数量。我们在数学、化学、生物、常识推理和人文学科共五个主流数据集上进行了验证，结果表明，VecCISC将总令牌使用量降低47%，同时保持或超越CISC的准确率。
原文摘要 · Abstract (English)
A standard technique for scaling inference-time reasoning is Self-Consistency, whereby multiple candidate answers are sampled from an LLM and the most common answer is selected. More recently, it has been shown that weighted majority voting (e.g. Confidence-Informed Self Consistency (CISC)), which assigns a confidence value to each candidate answer and chooses the answer with the largest accumulated score, tends to be more accurate on a wide range of popular benchmarks. In practice, weighted majority voting necessitates calling a critic LLM on each candidate's reasoning trace to produce the answer's confidence score. This secondary series of LLM calls greatly increases the overhead and cost of weighted majority voting, despite its potential performance benefits. To reduce this expense, we propose VecCISC, a lightweight, adaptive framework that uses a measure of semantic similarity to filter reasoning traces that are semantically equivalent to others, degenerate, or hallucinated, thus decreasing the number of candidate answers that must be evaluated by the critic. To ensure adequate experimental thoroughness, we evaluate VecCISC on five challenging, widely-adopted datasets spanning the domains of mathematics, chemistry, biology, commonsense reasoning, and the humanities. Our results demonstrate that VecCISC reduces the total token usage by 47%, while maintaining or exceeding the accuracy of CISC.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。