为大模型值对齐设计独立模块，提升生成内容的稳定性与安全性。
Toward Stable Value Alignment: Introducing Independent Modules for Consistent Value Guidance
- 引入独立价值模块，将价值观存储在隔离空间中，避免干扰主干网络。
- 通过可学习的桥接标记实现显式行为引导，降低有害内容超70%。
- 适用于需要稳定安全输出的场景，如内容生成与对话系统。
将大语言模型与人类价值观对齐通常依赖于后训练或推理时的参数或表征空间操控。然而，关键问题在于：模型残差流高度动态，价值观以脆弱、低维属性存在，与一致表达所需稳定性不兼容。本文提出稳定价值引导变换器（SVGT），通过两个核心设计解决此问题：(1) 独立价值建模，在与主干分离的专用价值空间中保持规范性表征；(2) 显式行为引导，将这些稳定信号转化为可学习的隐式桥接标记（Bridge Tokens）。这些标记作为动态价值锚点，显式引导生成轨迹，在不破坏主干内部表征的前提下，确保跨多种情境下的一致性。在多个主干模型和安全基准上的实验表明，SVGT普遍使有害内容得分降低超过70%，同时保持生成流畅性，验证了架构化价值建模的有效性。代码已开源至 https://github.com/Clervils/SVGT.git。
原文摘要 · Abstract (English)
Aligning large language models (LLMs) with human values typically relies on post-training or inference-time steering that directly manipulates the backbone's parameters or representation space. However, a critical gap exists: the model's residual stream is highly dynamic, in which values exist as fragile, low-dimensional properties, inherently incompatible with the stability required for consistent value expression. In this paper, we propose the Stable Value Guidance Transformer (SVGT), which addresses this gap through an independent value module incorporating two key designs: (1) independent value modeling, maintaining normative representations in a dedicated value space isolated from the backbone, and (2) explicit behavioral guidance, transducing these stable signals into learnable latent Bridge Tokens. These tokens serve as dynamic value anchors to explicitly steer the generative trajectory, ensuring robust adherence across diverse contexts without disrupting the backbone's internal representations. Experiments across multiple backbones and safety benchmarks show that SVGT generally reduces harmful scores by over 70% while maintaining generation fluency, demonstrating the efficacy of architecturally grounded value modeling. Our code is available at https://github.com/Clervils/SVGT.git.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。