让大模型对齐可切换,用自然语言指令实时控制行为。
ECLIPTICA -- A Framework for Switchable LLM Alignment via CITA - Contrastive Instruction-Tuned Alignment
- 通过对比指令微调,实现对齐指令的动态调节。
- 在多个评测中效率达86.7%,显著优于传统方法。
- 适合需要灵活调整模型行为的应用场景。
大语言模型的对齐通常是静态的:训练完成后策略即冻结。传统方法如DPO、GRPO通常将单一行为固化于权重中,运行时仅能通过提示技巧或昂贵的重新对齐进行调控。我们提出ECLIPTICA框架,将对齐视为受指令驱动且可运行时控制的行为:自然语言对齐指令作为显式的角色契约(立场、拒绝边界、表达程度),可在安全要求、用户角色和治理约束变化时实时调节模型行为。我们引入CITA(对比指令微调对齐),结合SFT与基于参考模型几何锚点的对比偏好优化,构建稳定的黎曼坐标系,确保指令更新保持在共享邻域内,使不同模式间可平稳切换。为分离对齐切换与普通指令遵循,我们发布ECLIPTICA基准:3000个受控案例(300个提示 × 10种指令类型),其中用户请求固定,仅对齐指令变化。在Llama-3.1-8B上,跨五个评测套件(ECLIPTICA, TruthfulQA, Conditional Safety, Length Control, LITMUS)中,CITA达到86.7%的指令对齐效率,显著超越DPO(56.1%)、GRPO(36.1%)和PPO(20.4%)。
原文摘要 · Abstract (English)
Alignment in large language models (LLMs) is still largely static: after training, the policy is frozen. DPO, GRPO methods typically imprint one behavior into the weights, leaving little runtime control beyond prompt hacks or expensive re-alignment. We introduce ECLIPTICA, which treats alignment as instruction-driven and runtime-controllable: natural-language alignment instructions act as an explicit behavioral contract (stance, refusal boundary, verbosity) that modulates behavior on the fly under evolving safety requirements, user roles, and governance constraints. We introduce CITA (Contrastive Instruction-Tuned Alignment), combining SFT with contrastive preference optimization under an explicit geometric anchor to a reference model. This yields a stable Riemannian chart and keeps instruction updates within a shared neighborhood, so regimes stay nearby and traversable for reliable switching. To isolate policy switching from ordinary instruction following, we release the ECLIPTICA benchmark: 3000 controlled cases (300 prompts x 10 instruction types) where the user request is fixed and only the alignment instruction changes. On Llama-3.1-8B across five suites (ECLIPTICA, TruthfulQA, Conditional Safety, Length Control, LITMUS), CITA reaches 86.7% instruction-alignment efficiency, beating DPO (56.1%), GRPO (36.1%), and PPO (20.4%).
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。