通过置信区间量化答案可靠性，提升大模型对齐的鲁棒性与效率
Conformal Feedback Alignment: Quantifying Answer-Level Reliability for Robust LLM Alignment
- 用置信预测构建答案可靠性区间，实现答案级不确定性建模
- 在多个数据集上提升对齐效果，减少对标注数据的依赖
- 适合关注模型可靠性评估与高效对齐的研究者
基于偏好对齐（如人类反馈强化学习，RLHF）依赖成对偏好进行训练，但标签常存在噪声和不一致。现有方法虽对偏好加权以应对不确定性，却忽略了更根本的问题：被比较答案本身的可靠性。为此，本文提出共形反馈对齐（CFA），将偏好加权建立在共形预测（CP）的统计保证基础上。CFA通过构建可控制覆盖率的共形预测集，量化每个答案的可靠性，并将这些可靠性信息整合为适用于DPO与PPO风格训练的合理权重。在多个数据集上的实验表明，CFA显著提升了对齐的鲁棒性与数据效率，证明建模答案侧不确定性可有效补充偏好层面加权，实现更稳健、更高效的大模型对齐。代码已公开。
原文摘要 · Abstract (English)
Preference-based alignment like Reinforcement Learning from Human Feedback (RLHF) learns from pairwise preferences, yet the labels are often noisy and inconsistent. Existing uncertainty-aware approaches weight preferences, but ignore a more fundamental factor: the reliability of the \emph{answers} being compared. To address the problem, we propose Conformal Feedback Alignment (CFA), a framework that grounds preference weighting in the statistical guarantees of Conformal Prediction (CP). CFA quantifies answer-level reliability by constructing conformal prediction sets with controllable coverage and aggregates these reliabilities into principled weights for both DPO- and PPO-style training. Experiments across different datasets show that CFA improves alignment robustness and data efficiency, highlighting that modeling \emph{answer-side} uncertainty complements preference-level weighting and yields more robust, data-efficient alignment. Codes are provided here.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。