用可学习嵌入选最语义一致的回答，提升长短问答可靠性。
Latent Self-Consistency for Reliable Majority-Set Selection in Short- and Long-Answer Reasoning
- 通过可学习的词元嵌入衡量语义一致性，选择最优答案。
- 在6个短答和5个长答数据集上均优于现有方法，误差低。
- 无需修改模型结构，推理开销不足1%，适合实际部署。
大语言模型的概率解码常导致复杂或长文本问题输出不一致。自一致性（SC）通过精确字符串投票缓解短问答问题，而通用自一致性（USC）与加权单字一致性分数（WUCS）虽扩展至长回答，但在短问答上精度下降。本文提出隐式自一致性（LSC），利用可学习的词元嵌入选择语义最一致的回答。LSC仅对摘要词元进行轻量级前向计算，相较基础LLM标准解码，运行时开销最多增加0.9%，且无需修改模型架构。在6个短问答与5个长问答推理基准（如MATH、MMLU、TruthfulQA）上，LSC在平均性能上超越SC、USC与WUCS，同时保持极低的预期校准误差，适用于多种回答格式。该方法兼具高准确率与良好置信度估计。
原文摘要 · Abstract (English)
Probabilistic decoding in Large Language Models (LLMs) often yields inconsistent outputs, particularly on complex or long-form questions. Self-Consistency (SC) mitigates this for short-form QA by majority voting over exact strings, whereas Universal Self-Consistency (USC) and Weighted Unigram Consistency Score (WUCS) extend to long-form responses but lose accuracy on short-form benchmarks. We introduce \textbf{Latent Self-Consistency (LSC)}, which selects the most semantically consistent response using learnable token embeddings. LSC's lightweight forward processing of summary tokens only introduces negligible runtime overhead (at most $0.9\%$) on top of standard decoding of the base LLM, and requires no changes to the model architecture. Across 6 short-form and 5 long-form reasoning benchmarks (e.g., MATH, MMLU, TruthfulQA), LSC surpasses SC, USC, and WUCS on both short-form and long-form on average performance, while adding negligible computational overhead on vanilla inference. These results position LSC as a reliable consistency-selection method that works effectively across various answer formats. Additionally, LSC provides well-calibrated confidence estimates, maintaining low expected calibration error across both answer formats.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。