通过分组集成提升大模型多选题判断的置信度准确性
Self-ensemble: Mitigating Confidence Mis-calibration for Large Language Models
- 将选项分组后跨组集成预测结果，改进置信度
- 在三个数据集上显著提升正确率，缓解过/欠校准问题
- 无需标注数据即可部署，适合现有模型快速升级
尽管大型语言模型（LLMs）在通用任务中表现良好，但在多选题问答（MCQA）任务中存在置信度失真问题，尤其当选项数量增加时更为严重。具体表现为：对正确答案过度保守，对错误答案过于自信，导致性能显著下降。为此，本文提出Self-ensemble方法：将候选答案分组，通过跨组集成模型预测结果做出最终决策。该方法具有即插即用特性，仅需设计注意力掩码与位置编码即可融入现有LLM架构，无需标注数据进行参数调优。在三个LLM与数据集上的实验表明，Self-ensemble全面缓解了置信度失真问题，优于标准推理及基线方法。
原文摘要 · Abstract (English)
Although Large Language Models (LLMs) perform well in general fields, they exhibit a confidence distortion problem on multi-choice question-answering (MCQA), particularly as the number of answer choices increases. Specifically, on MCQA with many choices, LLMs suffer from under-confidence in correct predictions and over-confidence in incorrect ones, leading to a substantially degraded performance. To solve this problem, we propose Self-ensemble in this work. Our method splits the choices into several groups and ensembles LLM predictions across these groups to reach a final decision. The advantage of Self-ensemble is its plug-and-play nature, where it can be integrated into existing LLM architecture based on a designed attention mask and positional encoding, without requiring labeled datasets for parameter tuning. Experimental results on three LLMs and datasets demonstrate that Self-ensemble comprehensively addresses the confidence distortion problem of LLMs, outperforming standard inference as well as baseline methods.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。