通过插入控制语句提升大模型在长对话中的对齐稳定性
LLM Reinforcement in Context
- 在用户输入中每隔x个词插入控制句，动态干预模型推理
- 实验证明该方法显著降低大模型越狱概率，尤其在长对话场景
- 适合关注安全对齐、提示工程与长程推理的从业者
当前大型语言模型对齐研究主要聚焦于通过示例和提示来增强模型对对抗攻击和异常行为的鲁棒性。研究表明，用户输入长度或对话轮次增加会提高大模型越狱的概率。然而，尚缺乏能随输入长度扩展的对齐强化方法。本文提出引入‘中断’机制作为解决方案：在用户输入中每隔x个词插入控制语句。我们建议该方法可推广至思维链（Chain-of-Thought）过程，以防止模型生成潜在恶意策略，从而在长上下文场景中维持更强的对齐性。
原文摘要 · Abstract (English)
Current Large Language Model alignment research mostly focuses on improving model robustness against adversarial attacks and misbehavior by training on examples and prompting. Research has shown that LLM jailbreak probability increases with the size of the user input or conversation length. There is a lack of appropriate research into means of strengthening alignment which also scale with user input length. We propose interruptions as a possible solution to this problem. Interruptions are control sentences added to the user input approximately every x tokens for some arbitrary x. We suggest that this can be generalized to the Chain-of-Thought process to prevent scheming.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。