用短后缀优化让大模型自适应禁用特定敏感词，无需重训练。
Adaptive Content Restriction for Large Language Models via Suffix Optimization
- 在提示词末尾添加可优化的短后缀，实现轻量级内容限制。
- 在80个敏感词上平均禁用率比系统级方法提升6%~17%。
- 适用于实际部署场景，对多个主流模型和线上平台有效。
大语言模型在各类应用中表现卓越，但强制内容限制仍具挑战性，因其输出空间庞大。传统对齐方法如监督微调（SFT）虽能防止生成有害内容，但不同用户群体的需求差异大、变化快，且与通用有害定义不一致，导致为每种场景单独微调不切实际。为此，我们提出新任务「自适应内容限制」（AdaCoRe），聚焦无需模型微调的轻量策略。本文提出首个方法「后缀优化」（SOP），通过向任意提示词添加一个短而优化的后缀，既能阻止目标模型生成指定受限词汇，又能保持输出质量。为评估该任务，我们构建了包含400个提示、80个受限词的「内容限制基准测试」（CoReBench），涵盖8个精心挑选类别。实验表明，SOP在Gemma2-2B、Mistral-7B、Vicuna-7B、Llama3-8B和Llama3.1-8B上分别较系统级基线提升15%、17%、10%、9%和6%的平均禁用率。同时，SOP在支持多种商业模型的POE平台也表现良好，验证了其实际部署可行性。
原文摘要 · Abstract (English)
Large Language Models (LLMs) have demonstrated significant success across diverse applications. However, enforcing content restrictions remains a significant challenge due to their expansive output space. One aspect of content restriction is preventing LLMs from generating harmful content via model alignment approaches such as supervised fine-tuning (SFT). Yet, the need for content restriction may vary significantly across user groups, change rapidly over time, and not always align with general definitions of harmfulness. Applying SFT to each of these specific use cases is impractical due to the high computational, data, and storage demands. Motivated by this need, we propose a new task called \textit{Adaptive Content Restriction} (AdaCoRe), which focuses on lightweight strategies -- methods without model fine-tuning -- to prevent deployed LLMs from generating restricted terms for specific use cases. We propose the first method for AdaCoRe, named \textit{Suffix Optimization (SOP)}, which appends a short, optimized suffix to any prompt to a) prevent a target LLM from generating a set of restricted terms, while b) preserving the output quality. To evaluate AdaCoRe approaches, including our SOP, we create a new \textit{Content Restriction Benchmark} (CoReBench), which contains 400 prompts for 80 restricted terms across 8 carefully selected categories. We demonstrate the effectiveness of SOP on CoReBench, which outperforms the system-level baselines such as system suffix by 15\%, 17\%, 10\%, 9\%, and 6\% on average restriction rates for Gemma2-2B, Mistral-7B, Vicuna-7B, Llama3-8B, and Llama3.1-8B, respectively. We also demonstrate that SOP is effective on POE, an online platform hosting various commercial LLMs, highlighting its practicality in real-world scenarios.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。