arXiv:2609.07808cs.CLcs.AI2026-09

大模型难以表达强烈情绪,因训练数据缺乏极端情感样本。

You Can't Prefer Emotions You Don't Sample: Intensity Undershoot in DPO-Tuned LLMs

  • 用连续情绪坐标控制模型输出,发现实际情绪强度远低于请求值
  • 情绪强度增益仅0.26(愉悦度)和0.13(激活度),远低于理想值1
  • 提升候选样本极端性可显著改善表现,说明瓶颈在数据而非方法

要求语言模型以‘非常兴奋’方式回应时,其输出通常仅轻微增强情绪。我们量化该现象:将指令微调模型置于连续效价-唤醒(VA)目标下,效价衡量情绪愉悦程度,唤醒衡量激活水平,扫描请求值从-1到+1,使用冻结回归器测量实际情绪。结果发现,响应变化远小于请求:在Llama-3.1-8B上,效价增益为0.26,唤醒增益为0.13,理想控制器应为1。模型系统性低估情感强度,印证了Fazzi等人(2025)的定性观察。实验表明此问题源于偏好学习流程——自然语料(如EmoBank)以中性为主,候选样本极少达到极端情绪,导致直接偏好优化(DPO)无极端样本可选。若改用均匀覆盖目标空间并采样更多极端候选,效价增益升至0.40±0.02(3个种子),外推误差下降,仅小幅增加分布内成本(EmoBank测试集VA距离由0.092增至0.107)。相同方法在Qwen3-8B上重现(效价增益0.44,分布内准确率不变)。唤醒更难调控,增益平均变化甚微且波动大(0.14±0.07),因基础模型不愿生成高唤醒内容。证据表明,情感强度忠实度受限于候选池极端性,而非条件格式。

原文摘要 · Abstract (English)

Ask a language model to respond "very excitedly," and its output is typically only mildly more energetic. We quantify this effect. We condition an instruction-tuned LLM on a continuous Valence-Arousal (VA) target, where valence measures how pleasant a state is and arousal how activated it is, measure the achieved affect with a frozen regressor, and sweep the requested target from -1 to +1. The response moves far less than asked: the gain, the slope of achieved against requested affect, is only 0.26 for valence and 0.13 for arousal on Llama-3.1-8B, where a faithful controller would score 1. The model systematically undershoots requested emotional intensity, which puts a number on the qualitative observation of Fazzi et al. (2025). Our experiments trace this to the preference-learning pipeline. Training targets from natural corpora such as EmoBank are neutral-heavy, and the sampled candidates themselves rarely reach extreme affect, so Direct Preference Optimization (DPO) is left with no extreme exemplar to prefer. If instead we cover the target space uniformly and sample a hotter, larger candidate pool, valence gain rises from 0.26 to 0.40 +/- 0.02 (3 seeds) and extrapolation error drops, at only a modest in-distribution cost (EmoBank-test VA distance 0.092 to 0.107). The same recipe reproduces on Qwen3-8B (gain_v 0.44, with in-distribution accuracy preserved). Arousal is harder and less reliable: its gain barely moves on average and swings across seeds (0.14 +/- 0.07, against valence's tight +/- 0.02), because raising arousal needs candidates the base model is reluctant to generate. The evidence indicates that faithful intensity is bottlenecked by the extremity of the candidate pool rather than by the conditioning format.

情感控制偏好学习大模型行为情绪生成

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。