用自动机引导的束搜索,让生成文本100%满足规则约束。
ABS: Enforcing Constraint Satisfaction On Generated Sequences Via Automata-Guided Beam Search
- 通过有限状态自动机约束束搜索路径,实时屏蔽违规生成。
- 在图像流分类、文本生成等任务中实现100%约束满足率。
- 无需重训练,适用于所有可编译为DFA的规则,适合需合规输出场景。
序列生成与预测是现代机器学习的核心任务,广泛应用于自然语言处理、程序合成和时间序列预测等领域。这些任务通常采用自回归建模,每次生成一个标记并基于前序内容进行条件预测,束搜索常用于平衡探索与流畅性。尽管深度学习模型和大语言模型在此类设置下擅长捕捉统计模式,但难以保证形式化约束的遵守。本文提出ABS:一种通用且模型无关的推理时算法,可确保任何可编译为确定性有限自动机(DFA)的约束得到满足,且无需重新训练。ABS利用DFA引导一种受约束的束搜索:在每个解码步骤中,屏蔽导致违反约束的转移路径,同时根据模型概率与自动机接受结构对剩余路径进行动态重排序。我们形式化证明了生成序列必然满足给定约束,并实证表明ABS还能提升输出质量。我们在三个不同任务上验证该方法:受限图像流分类、可控文本生成和文本补全。在所有设置中,ABS均实现100%的约束满足率,同时在标准质量指标和效率上优于或匹配现有最佳基线。
原文摘要 · Abstract (English)
Sequence generation and prediction form a cornerstone of modern machine learning, with applications spanning natural language processing, program synthesis, and time-series forecasting. These tasks are typically modeled in an autoregressive fashion, where each token is generated conditional on the preceding ones, and beam search is commonly used to balance exploration and fluency during decoding. While deep learning models and Large Language Models (LLMs) excel at capturing statistical patterns in this setting, they remain ill-equipped to guarantee compliance with formal constraints. In this paper, we introduce ABS: a general and model-agnostic inference-time algorithm that guarantees compliance with any constraint that can be compiled into a Deterministic Finite Automaton (DFA), without requiring retraining. ABS leverages the DFA to guide a constrained variant of beam search: at each decoding step, transitions leading to violations are masked, while remaining paths are dynamically re-ranked according to both the model's probabilities and the automaton's acceptance structure. We formally prove that the resulting sequences are guaranteed to satisfy the given constraints, and we empirically demonstrate that ABS also improves output quality. We validate our approach on three distinct tasks: constrained image-stream classification, controlled text generation, and text infilling. In all settings, ABS achieves perfect constraint satisfaction, while outperforming or matching state-of-the-art baselines on standard quality metrics and efficiency.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。