通过优化输入词嵌入，实现对安全模型输出的实时可控调节。
Test-Time Safety Alignment
- 用零阶梯度估计黑箱内容审核API，微调输入词嵌入
- 在标准安全基准上使所有被标记有害的输出归零
- 适用于需实时安全控制的生成式模型场景
近期研究发现，模型的输入词嵌入可作为有效控制变量，引导其输出满足特定属性。但此前仅在预训练文本续写模型上验证，且目标仅为降低短续写中的表面粗俗。一个自然且实际重要的问题是：输入嵌入能否控制产生非平衡双模拒绝-服从分布的安全对齐模型？我们在此背景下探索安全控制，发现可通过亚词汇级优化输入嵌入，最小化对齐模型输出的语义危害性。方法利用黑箱文本审核API的零阶梯度估计，对输入嵌入进行梯度下降以降低生成文本的危害性。实验表明，该方法可在标准安全基准上中和所有被安全标记的响应。
原文摘要 · Abstract (English)
Recent work has shown that a model's input word embeddings can serve as effective control variables for steering its behavior toward outputs that satisfy desired properties. However, this has only been demonstrated for pretrained text-completion models on the relatively simple objective of reducing surface-level profanity in short continuations. A natural and practically important question is how well input embeddings can control aligned models, which produce an imbalanced bimodal refuse-or-comply output distribution rather than the smooth distribution characteristic of open-ended generation. We explore this in the context of safety, showing that input word embeddings can be optimized in a sub-lexical manner to minimize the semantic harmfulness of aligned model responses. Our approach uses zeroth-order gradient estimation of a black-box text-moderation API with respect to the input embeddings, and then applies gradient descent on these embeddings to minimize the harmfulness of the generated text. Experiments show that the proposed method can neutralize every safety-flagged response on standard safety benchmarks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。