让大模型在无标注情况下也能讲清推理过程。
Anchored Alignment for Self-Explanations Enhancement
- 用三类输出分组优化偏好对选择，提升模型自解释能力。
- 相比其他微调方法，解释质量显著提升且准确率不变。
- 适合需要可解释AI的科研与工业落地场景。
本文提出一种用于增强大语言模型自解释能力的对齐方法，无需依赖标注的推理说明即可提升模型推理过程的可解释性。该方法包含三个核心组件：解释质量评估、自指令数据集生成和模型对齐。此外，我们提出一种基于锚点偏好对的对齐技术，将模型输出分为三类：始终正确、始终错误和结果波动。针对每类采用不同策略，从而提升直接偏好优化（DPO）的效果。实验表明，该方法在保持准确率的同时，显著改善了解释质量，优于其他微调策略。
原文摘要 · Abstract (English)
In this work, we introduce a methodology for alignment designed to enhance the ability of large language models (LLMs) to articulate their reasoning (self-explanation) even in the absence of annotated rationale explanations. Our alignment methodology comprises three key components: explanation quality assessment, self-instruction dataset generation, and model alignment. Additionally, we present a novel technique called Alignment with Anchor Preference Pairs, which improves the selection of preference pairs by categorizing model outputs into three groups: consistently correct, consistently incorrect, and variable. By applying tailored strategies to each category, we enhance the effectiveness of Direct Preference Optimization (DPO). Our experimental results demonstrate that this approach significantly improves explanation quality while maintaining accuracy compared to other fine-tuning strategies.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。