对齐后的大模型回答趋于单一，影响不确定性估计效果。
The Alignment Tax: Response Homogenization in Aligned LLMs and Its Implications for Uncertainty Estimation
- 通过分析10次独立采样发现40%-79%问题生成同一语义回答簇。
- 基于采样的不确定性方法在受影响问题上失效（AUROC=0.500），但词元熵仍有效（0.603）。
- 该现象由DPO训练阶段引发，适用于不同模型家族和任务，适合关注可靠性评估的研究者。
经过强化学习人类反馈（RLHF）对齐的语言模型表现出回答同质化：在TruthfulQA数据集（n=790）上，40%-79%的问题在10次独立采样中产生单一语义簇。在这些受影问题上，基于采样的不确定性方法判别能力归零（AUROC=0.500），而自由词元熵仍保留信号（0.603）。该对齐税具有任务依赖性：在GSM8K（n=500）上，词元熵达到0.724（Cohen's d=0.81）。基础模型与指令微调模型对比证实对齐的因果作用：基础模型单簇率仅1.0%，指令模型达28.5%（p < 10^{-6}）。训练阶段消融实验（基础模型0.0% → SFT 1.5% → DPO 4.0% SCR）将根源定位至DPO而非SFT。在四个模型家族中跨族验证显示对齐税严重程度随家族与规模变化。22项实验、5个基准、3种模型规模（3B-14B）及三种DeBERTa嵌入规模（均~0.51 AUROC）下，采用杰卡德、嵌入与NLI基线验证结果稳健。使用两个独立嵌入族进行交叉验证排除耦合偏差。在WebQuestions上（58.0% SCR）的跨数据集验证确认其泛化能力。核心发现——回答同质化——为实现无关且无标签。基于此诊断，我们提出最便宜优先级级联（UCBD）策略，融合正交不确定性信号。选择性预测使GSM8K准确率从84.4%提升至93.2%（50%覆盖率）；弱相关边界（|r| ≤ 0.12）实现57%成本节省。
原文摘要 · Abstract (English)
RLHF-aligned language models exhibit response homogenization: on TruthfulQA (n=790), 40-79% of questions produce a single semantic cluster across 10 i.i.d. samples. On affected questions, sampling-based uncertainty methods have zero discriminative power (AUROC=0.500), while free token entropy retains signal (0.603). This alignment tax is task-dependent: on GSM8K (n=500), token entropy achieves 0.724 (Cohen's d=0.81). A base-vs-instruct ablation confirms the causal role of alignment: the base model shows 1.0% single-cluster rate vs. 28.5% for the instruct model (p < 10^{-6}). A training stage ablation (Base 0.0% -> SFT 1.5% -> DPO 4.0% SCR) localizes the cause to DPO, not SFT. Cross-family replication on four model families reveals alignment tax severity varies by family and scale. We validate across 22 experiments, 5 benchmarks, 4 model families, and 3 model scales (3B-14B), with Jaccard, embedding, and NLI-based baselines at three DeBERTa scales (all ~0.51 AUROC). Cross-embedder validation with two independent embedding families rules out coupling bias. Cross-dataset validation on WebQuestions (58.0% SCR) confirms generalization beyond TruthfulQA. The central finding -- response homogenization -- is implementation-independent and label-free. Motivated by this diagnosis, we explore a cheapest-first cascade (UCBD) over orthogonal uncertainty signals. Selective prediction raises GSM8K accuracy from 84.4% to 93.2% at 50% coverage; weakly dependent boundaries (|r| <= 0.12) enable 57% cost savings.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。