用最小值选择解决多目标对齐失衡问题，让语言代理兼顾多个目标。
MINT: Min-Selection Preference Distillation for Balanced Multi-Objective Alignment
- 通过选取最弱目标得分最高的候选，实现均衡对齐。
- 情感支持任务中弱目标得分从0.37提升至0.64（p<10^-40）。
- 适合需要多目标平衡的对话系统与智能体训练场景。
在基于偏好的训练中，同时对齐多个目标时，常因目标加权求和导致优化偏向成本最低的项，牺牲其他目标，使支持型代理仅言辞温暖却无实质帮助。根本原因在于加性奖励缺乏平衡意识。本文提出MINT（Min-Selection Preference Distillation），仅需修改偏好蒸馏中的排序方式：不再按加权奖励和排序，而是按各目标中最差表现排序，从而蒸馏出最佳平衡的候选，且保持原始DPO目标不变。该方法对应广义平均族中p→-∞的极限，即最坏情况选择。在合作式情感支持与对抗式谈判任务中，最小值选择显著提升所有目标性能并大幅降低失衡度；在情感支持任务中，弱目标得分由0.37升至0.64（p < 10^-40），超越人类专家，并在全轮对话中持续有效。逐轮分析表明，该方法纠正失衡的程度与参考策略的不平衡程度成正比，其优势可持续到不平衡存在的整个交互周期。
原文摘要 · Abstract (English)
Aligning a language agent to several objectives at once is a persistent failure mode of preference-based training: when objectives are combined additively, optimization collapses onto whichever is cheapest to improve and sacrifices the rest, so a support agent learns to sound warm while giving no real help. The root issue is that an additive reward has no notion of balance. We introduce Mint (MIN-selection preference disTillation), a one-line change to preference distillation: rather than ranking sampled candidates by a weighted sum of rewards, we rank them by their weakest objective, distilling the best-balanced candidate over the most lopsided one with an unchanged DPO objective. This is the p -> negative infinity limit of a generalized-mean family spanning additive to worst-case selection. Across cooperative emotional support and adversarial negotiation, min-selection lifts both objectives while sharply cutting their imbalance; on emotional support it raises the weaker axis from 0.37 to 0.64 (p < 10^-40), surpassing human experts and persisting across full multi-turn rollouts. A turn-by-turn analysis yields our central finding: min-selection corrects imbalance in proportion to how imbalanced the reference policy is, and its benefit endures over an interaction precisely as long as that imbalance does.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。