NCO让大模型生成时自动避开敏感内容,效率高且兼容主流推理方式。
NCO: A Versatile Plug-in for Handling Negative Constraints in Decoding

- 在线匹配硬规则与正则表达式,避免状态爆炸
- 支持多种采样和束搜索,计算开销小
- 可抑制敏感词、个人隐私信息等不良输出
控制大语言模型生成不良内容(如脏话、个人身份信息)日益重要。早期方法依赖后处理或重采样,近年转向生成过程中约束解码以降低计算成本和质量损失。但同时阻止多个禁止规则或正则表达式出现在输出中仍具挑战性。直接将所有规则合并为单一自动机虽可行,但常导致状态规模过大;标准正则引擎也缺乏构建此类约束所需的补集与交集操作。为此,我们提出NCO,一种在线模式匹配解码策略,在不引发状态爆炸的前提下,高效处理有限硬约束与正则约束。NCO完全兼容标准推理策略,包括各类采样方法与束搜索,并支持软掩码实现概率抑制。我们在实际任务中验证了其在保护个人身份信息与过滤脏话方面的有效性。代码已开源:https://github.com/hyundong98/NCO-Decoding.git。
原文摘要 · Abstract (English)
Controlling Large Language Models (LLMs) to prevent the generation of undesirable content, such as profanity and personally identifiable information (PII), has become increasingly critical. While earlier approaches relied on post-processing or resampling, recent research has shifted towards constrained decoding methods that control outputs during generation to mitigate high computational costs and quality degradation. However, preventing multiple forbidden hard constraints or regex constraints from appearing anywhere in the output is computationally challenging. A straightforward solution is to convert these constraints into a single automaton that tracks all forbidden patterns during decoding, but this often becomes impractically large. Standard regex engines also do not readily support the operations needed to build such a constraint, such as complement and intersection. In order to address these limitations, we propose NCO, a decoding strategy that performs online pattern matching over finite hard constraints and regex constraints, reducing computational overhead without inducing state explosion. NCO is fully compatible with standard inference strategies, including various sampling methods and beam search, while also supporting soft masking for probabilistic suppression. We empirically demonstrate its effectiveness across practical tasks, including PII and profanity suppression. Our implementation is available at https://github.com/hyundong98/NCO-Decoding.git .
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。