让大模型先自由构思，再强制语法正确，提升生成准确性。
The Hidden Cost of Structured Generation in LLMs: Draft-Conditioned Constrained Decoding
- 先自由生成草稿，再基于草稿约束解码，分离语义与结构。
- 在GSM8K上准确率提升24个百分点，10亿参数模型媲美更大模型。
- 无需训练，适合生成代码、JSON等需严格格式的场景。
大语言模型生成代码、JSON等可执行输出时，一个语法错误即导致无效结果。传统约束解码通过逐词掩码和重归一化保证结构合法，但当有效延续概率过低时，会导向局部合法却语义错误的路径。本文提出无训练、两步式的草稿-条件约束解码（DCCD）：先生成无约束草稿，再以草稿为条件进行约束解码，确保合法性。理论分析显示，草稿条件能增加可行概率质量，降低硬约束带来的累积“投影代价”，并支持最佳K个草稿选择。在结构化推理基准上，相比标准约束解码，DCCD将严格结构准确率最高提升24个百分点（如1B模型在GSM8K上从15.2%升至39.0%），并使小模型对齐甚至超越大模型基线，显著提升参数效率。
原文摘要 · Abstract (English)
Large language models (LLMs) are increasingly used to generate executable outputs, JSON objects, and API calls, where a single syntax error can make the output unusable. Constrained decoding enforces validity token-by-token via masking and renormalization, but it can distort generation when the model assigns low probability mass to valid continuations, pushing decoding toward locally valid yet semantically incorrect trajectories. We propose \emph{Draft-Conditioned Constrained Decoding (DCCD)}, a simple two-step, training-free inference procedure that decouples semantic planning from structural enforcement: an unconstrained draft is generated first, and constrained decoding is then applied, conditioned on this draft, to guarantee validity. We analyze DCCD through a KL-projection view, showing that draft conditioning increases feasible mass and reduces the cumulative "projection tax" induced by hard constraints, with an optional best-of-$K$ draft selection. Across structured reasoning benchmarks, DCCD improves strict structured accuracy by up to +24 percentage points over standard constrained decoding (e.g., 15.2\% to 39.0\% on GSM8K with a 1B model), and enables smaller model pairs to match or exceed much larger constrained baselines, yielding substantial gains in parameter efficiency.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。