小模型生成结构化输出常出错，新方法让结果又准又合规。
When Correct Isn't Usable: Improving Structured Output Reliability in Small Language Models
- 用迭代优化系统提示词，不需微调即可提升输出合规性。
- 在GSM8K上达84%-87%准确率，MATH上34%-40%，远超传统提示。
- 对闭源大模型也有效，且延迟几乎无增加，适合实际部署场景。
部署的语言模型必须产出既正确又符合格式的输出。本文以GSM8K和MATH两个数学基准测试为受控实验平台，研究结构化输出可靠性差距：真实答案明确，输出格式要求严格（含必填字段的JSON）。评估三个7-9B参数规模模型在五种提示策略下的表现，以任务准确率（数学正确性与有效JSON结构的联合事件）为主要指标。发现系统性格式失败：NAIVE提示（无系统提示）在GSM8K上任务准确率达85%，但所有模型和数据集上输出准确率均为0%；REFERENCE提示（最小手写JSON格式提示）对其中两个模型仍导致0%输出准确率。约束解码虽能保证语法正确，但引入3.6倍至8.2倍延迟开销，且部分情况下显著降低任务性能。为此，我们开发AloLab，一个仅需黑盒API访问目标模型的迭代系统提示优化器（元智能体：Claude Sonnet 4.5），在五次独立运行中实现GSM8K上84%-87%、MATH上34%-40%的输出准确率，30组配对McNemar检验中有29组优于最佳静态提示（p < 0.05），且接近NAIVE推理延迟，无需模型微调。该格式失效现象同样存在于GPT-4o（OpenAI, 2024）：REFERENCE提示因系统性markdown代码块包裹导致0%输出准确率，而AloLab达到95.2% [94.8, 95.6]。消融实验显示，若将元智能体替换为Claude 3 Haiku，平均输出准确率降至61.0%，运行间标准差从<1个百分点升至21.8个百分点，确认元智能体能力是优化质量的关键驱动因素。
原文摘要 · Abstract (English)
Deployed language models must produce outputs that are both correct and format-compliant. We study this structured-output reliability gap using two mathematical benchmarks -- GSM8K and MATH -- as a controlled testbed: ground truth is unambiguous and the output contract is strict (JSON with required fields). We evaluate three 7-9B models under five prompting strategies and report output accuracy -- the joint event of mathematical correctness and valid JSON structure -- as the primary metric. A systematic format failure emerges: NAIVE prompting (no system prompt) achieves up to 85% task accuracy on GSM8K but 0% output accuracy across all models and datasets. REFERENCE prompting (a minimal hand-written JSON format prompt) fares little better, yielding 0% output accuracy for two of four models tested. Constrained decoding enforces syntactic validity but incurs 3.6x-8.2x latency overhead and in several settings degrades task performance substantially. To overcome this limitation, we developed AloLab, an iterative system-prompt optimizer (meta-agent: Claude Sonnet 4.5) requiring only black-box API access to the target model; it reaches 84-87% output accuracy on GSM8K and 34-40% on MATH across five independent runs per model, with 29/30 paired McNemar comparisons against the best static prompt significant at p < 0.05, at near-NAIVE inference latency and without model fine-tuning. The same format failure extends to GPT-4o (OpenAI, 2024), a proprietary closed-source model: REFERENCE achieves 0% output accuracy due to systematic markdown-fence wrapping, while AloLab reaches 95.2% [94.8, 95.6]. An ablation replacing the Sonnet 4.5 meta-agent with Claude 3 Haiku reduces mean output accuracy to 61.0% and increases run-to-run standard deviation from <1 pp to 21.8 pp, confirming that meta-agent capability is a primary driver of optimization quality.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。