arXiv:2606.13227cs.CL2026-06

让AI对话更自然,按语境匹配人类真实回答风格

PolyAlign: Conditional Human-Distribution Alignment

论文配图:PolyAlign: Conditional Human-Distribution Alignment
图 1 · 摘自论文原文
  • 按语言、对话类型等分桶构建人类回复分布,精准对齐
  • 在中英文多轮对话中,自然度提升12.3%,分布契合度提高18%
  • 适合需要本地化、情境化对话的AI应用开发者

后训练方法如监督微调(SFT)和偏好优化通常将语言模型对齐到单一全局助手行为,虽能提升平均帮助性,却抑制了跨语言、任务和对话场景下人类回复的自然多样性。本文将此问题定义为条件性人类分布对齐:模型应匹配当前交互情境下的恰当人类回复分布,而非统一风格。提出PolyAlign框架,将双语交互数据按语言、对话轨迹、回复家族和长度划分为特定桶,构建桶内人类参考分布。结合桶感知SFT与人类分布偏好优化(HDPO),后者利用评分类估计与桶内人类支持集的距离进行正则化。在涵盖英中文单轮与多轮设置的双语评估套件中,PolyAlign在保持竞争力任务性能的同时,显著提升条件自然度与分布忠实性。结果表明,后训练应从全局对齐转向面向交互情境的人类分布对齐。

原文摘要 · Abstract (English)

Post-training methods such as supervised fine-tuning (SFT) and preference optimization typically align language models toward a single global assistant behavior. While effective for improving average helpfulness, this can suppress the natural variation of human responses across languages, tasks, and dialogue settings. We study this problem as conditional human-distribution alignment: models should match the human response distribution appropriate to the current interaction context, rather than a universal response style. We introduce PolyAlign, a distribution-aware alignment framework that organizes bilingual interaction data into bucket-specific human reference distributions defined by language, interaction track, response family, and length. PolyAlign combines Bucket-Aware SFT, which balances optimization across heterogeneous buckets, with Human-Distribution Preference Optimization (HDPO), which regularizes preference learning using critic-estimated distance to bucket-specific human support. Across a bilingual evaluation suite covering English and Chinese single- and multi-turn settings, PolyAlign improves conditional naturalness and distributional faithfulness while preserving competitive task utility. The results suggest that post-training should move beyond global alignment objectives toward interaction-aware alignment with human response distributions.

对话系统分布对齐多语言SFT

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。