让大模型在辩论中坚持正确答案，不被错误说法带偏。
Aligning Large Language Models for Faithful Integrity Against Opposing Argument
- 用双向置信度估计衡量模型对问题和答案的自信程度。
- 在多个基准上显著提升模型面对反驳时保持正确性的能力。
- 适合需要高可信度对话系统的研究者与开发者使用。
大型语言模型在复杂推理任务中表现出色，但在对话中容易被不忠实的论点误导，即使其初始陈述正确。为此，本文研究如何保持大模型在对话中的忠实完整性，即在面对反对观点时坚持正确回答，并在遇到可靠论点时修正错误。我们提出一种名为基于置信度估计的忠实一致性对齐（AFICE）的新框架。AFICE首先设计双边置信度估计（BCE）方法，通过解码过程中的内部状态和累积概率比，分别估计模型对问题和答案的置信度。基于BCE构建包含上下文、原始陈述和反驳论点的对话偏好数据集，利用直接偏好优化（DPO）对模型进行对齐训练。大量实验结果表明，该方法显著提升了模型在遭遇反对意见时维持忠实回应的能力，增强了大模型在复杂交互场景中的实用性与可信度。代码与数据将通过https://github.com/zhaoy777/AFICE.git发布。
原文摘要 · Abstract (English)
Large Language Models (LLMs) have demonstrated impressive capabilities in complex reasoning tasks. However, they can be easily misled by unfaithful arguments during conversations, even when their original statements are correct. To this end, we investigate the problem of maintaining faithful integrity in LLMs. This involves ensuring that LLMs adhere to their faithful statements in the face of opposing arguments and are able to correct their incorrect statements when presented with faithful arguments. In this work, we propose a novel framework, named Alignment for Faithful Integrity with Confidence Estimation (AFICE), which aims to align the LLM responses with faithful integrity. Specifically, AFICE first designs a Bilateral Confidence Estimation (BCE) approach for estimating the uncertainty of each response generated by the LLM given a specific context, which simultaneously estimate the model's confidence to the question based on the internal states during decoding as well as to the answer based on cumulative probability ratios. With the BCE, we construct a conversational preference dataset composed of context, original statement, and argument, which is adopted for aligning the LLM for faithful integrity using Direct Preference Optimization (DPO). Extensive experimental results on a wide range of benchmarks demonstrate significant improvements in the LLM's ability to maintain faithful responses when encountering opposing arguments, ensuring both the practical utility and trustworthiness of LLMs in complex interactive settings. Code and data will be released via https://github.com/zhaoy777/AFICE.git
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。