arXiv:2510.01243cs.CL2025-10NeurIPS被引 8

通过建模隐空间毒性过渡,实现大模型生成内容的精准去毒。

Detoxifying Large Language Models via Autoregressive Reward Guided Representation Editing

  • 在隐空间构建毒性和非毒性间的精细过渡轨迹,指导去毒编辑。
  • 相比基线,毒性降低62.21%,推理时间减少47.58%。
  • 适用于需要安全部署的大模型,尤其适合测试时快速干预。

大型语言模型(LLMs)在各类任务中表现卓越,但仍易生成有害内容,亟需去毒策略以保障安全部署。测试时去毒方法通过在模型表示中引入静态或动态干预,具有灵活性和低侵入性优势。然而,现有方法因未能充分探索毒性与非毒性输出间的转换空间,常导致干预不精准。为此,我们提出自回归奖励引导表示编辑框架ARGRE,显式建模隐空间中的毒性转变路径,实现稳定精确的奖励引导编辑。ARGRE识别非毒性语义方向,通过毒-非毒表示插值揭示细粒度转变轨迹,将稀疏毒性标注转化为密集训练信号,构建自回归奖励模型以提供稳定编辑指引。推理阶段,该模型引导两步自适应编辑:首先基于预期奖励差进行方向性引导,将表示移向非毒性区域;随后执行轻量梯度优化。在8个主流大模型上的实验证明,ARGRE显著优于领先基线,在有效性上提升62.21%(毒性降低),效率提升47.58%(推理时间减少),同时对原始模型核心能力影响极小。

原文摘要 · Abstract (English)

Large Language Models (LLMs) have demonstrated impressive performance across various tasks, yet they remain vulnerable to generating toxic content, necessitating detoxification strategies to ensure safe and responsible deployment. Test-time detoxification methods, which typically introduce static or dynamic interventions into LLM representations, offer a promising solution due to their flexibility and minimal invasiveness. However, current approaches often suffer from imprecise interventions, primarily due to their insufficient exploration of the transition space between toxic and non-toxic outputs. To address this challenge, we propose \textsc{A}utoregressive \textsc{R}eward \textsc{G}uided \textsc{R}epresentation \textsc{E}diting (ARGRE), a novel test-time detoxification framework that explicitly models toxicity transitions within the latent representation space, enabling stable and precise reward-guided editing. ARGRE identifies non-toxic semantic directions and interpolates between toxic and non-toxic representations to reveal fine-grained transition trajectories. These trajectories transform sparse toxicity annotations into dense training signals, enabling the construction of an autoregressive reward model that delivers stable and precise editing guidance. At inference, the reward model guides an adaptive two-step editing process to obtain detoxified representations: it first performs directional steering based on expected reward gaps to shift representations toward non-toxic regions, followed by lightweight gradient-based refinements. Extensive experiments across 8 widely used LLMs show that ARGRE significantly outperforms leading baselines in effectiveness (-62.21% toxicity) and efficiency (-47.58% inference time), while preserving the core capabilities of the original model with minimal degradation. Our code is available at the website.

去毒大模型奖励模型表示编辑

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。