让大模型实时适应社会规范变化，通过动态调整提示实现安全可控推理。
Steering Frozen LLMs: Adaptive Social Alignment via Online Prompt Routing
- 基于在线提示路由的自适应对齐框架，无需重新训练。
- 在实验中提升10.98%累积奖励，降低14.42%次优差距。
- 适合需要持续安全更新的部署场景，如对话系统与内容审核。
大型语言模型通常通过后训练对齐（如RLHF或DPO）进行治理，导致部署阶段策略固定不变。然而，真实世界的安全问题具有全生命周期特性：静态防御难以应对不断演化的越狱行为，固定权重也无法适应多元、动态变化的安全规范。这促使我们在推理阶段实现治理，以不依赖昂贵重训的方式引导模型行为。为此，我们提出一致聚类线性上下文赌博机（CCLUB），一种通过系统级提示路由实现自适应社会对齐的统一框架。CCLUB采用保守的一致性聚类机制：仅在效用与安全相似性图的交集内聚合数据，有效防止语义相近但风险相异情境间的不安全泛化。理论分析表明其具备亚线性遗憾上界，证明了近最优性能。大量实验证明，相比强基线，CCLUB在累积奖励上提升10.98%，平均次优差距下降14.42%。
原文摘要 · Abstract (English)
Large language models (LLMs) are typically governed by post-training alignment (e.g., RLHF or DPO), which yields a largely static policy during deployment and inference. However, real-world safety is a full-lifecycle problem: static defenses degrade against evolving jailbreak behaviors, and fixed weights cannot adapt to pluralistic, time-varying safety norms. This motivates inference-time governance that steers behavior without costly retraining. To address this, we introduce the Consensus Clustering LinUCB Bandit (CCLUB), a unified framework for adaptive social alignment via system-prompt routing. CCLUB employs a conservative consensus clustering mechanism: it pools data only within the intersection of utility and safety similarity graphs, effectively preventing unsafe generalization across semantically proximal but risk-divergent contexts. Our theoretical analysis yields a sublinear regret guarantee, demonstrating near-optimal performance of CCLUB. Extensive experiments validate that CCLUB outperforms strong baselines, achieving a 10.98% improvement in cumulative reward and a 14.42% reduction in the average suboptimality gap.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。