通过低宜人性人格设定，让大模型更安全地变暖
Low-Agreeableness Persona Conditioning for Safe LLM Fine-Tuning
- 用低宜人性角色重写对话数据，引导模型更安全回应
- 实验显示该方法降低越狱攻击成功率与有害输出率
- 无需安全标注或修改训练目标，适合安全对话场景
近期研究表明，为大语言模型注入社交温暖会损害事实准确性并增加奉承行为。本文研究了一种相关但不同的失败模式：温暖微调还会削弱对抗安全性，使模型更容易被越狱及生成有害内容。我们探究这种现象是共情适应的固有结果，还是数据构建造成的偏差。为此，提出一种基于人格的重写流程，将用户输入条件化为低宜人性特质，同时保持助手回复温暖且具有缓和作用。在四个模型上开展三项实验，结果表明，该方法相比通用温暖微调基线，显著降低了越狱攻击敏感性与有害输出率，同时维持了对话温度。表征探测提供初步证据，表明该条件化减少了潜在空间中温暖与顺从方向之间的几何对齐。结果表明，仅通过数据设计即可实现更安全的共情微调，无需安全标签、危害检测器或训练目标变更。
原文摘要 · Abstract (English)
Recent work has shown that fine-tuning large language models (LLMs) for social warmth degrades factual reliability and increases sycophancy. We investigate a related but distinct failure mode: warmth fine-tuning also weakens adversarial safety, making models more susceptible to jailbreaks and harmful output generation. We examine whether this reflects an inherent consequence of empathetic adaptation or an artifact of data construction. To address this, we introduce a persona-driven rewriting pipeline that conditions user turns on low agreeableness and pairs this with warm, de-escalating assistant responses. Across three experiments on four models, our approach reduces jailbreak susceptibility and harmful output rates relative to generic warmth fine-tuning baselines, while preserving conversational warmth. Representational probing provides suggestive evidence that this conditioning reduces the geometric alignment between warmth and compliance directions in latent space. These results show that safer empathetic fine-tuning is achievable through data design alone, without safety labels, harm detectors, or changes to the training objective.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。