arXiv:2606.09410cs.AIcs.CL2026-06

模型能力不足时,格式化会严重拖垮推理效果。

Capacity, Not Format: Rethinking Structured Reasoning Failures

  • 用匹配信息量的自然语言对照实验,分离出格式影响与提示长度干扰。
  • 能力接近极限的模型在格式化时性能下降超28个百分点,且与令牌耗尽无关。
  • 建议先自由思考再格式化,尤其适用于资源受限的模型。

以往研究将结构化输出视为推理负担,但这一观点不完整:格式成本高度依赖模型的剩余能力。通过信息量匹配的自然语言控制和四级模式复杂度梯度,我们在4个模型和5个基准上分离了格式特异性影响与提示长度混淆因素,所有成功生成响应均无解析失败。结果表明,结构化格式的影响是容量依赖的。具备足够余量的模型(如Sonnet)在使用JSON格式时表现几乎不受影响(MATH-Hard上:88.7±4.0% vs. CoT 89.3±1.7%)。相反,接近能力极限的模型则因两种机制严重退化:其一,在标准令牌预算下,Haiku因截断下降36.2pp(p < 0.0001);其二,即使延长预算消除截断,GPT-4o-mini仍下降28.0pp(p < 0.001),揭示了纯容量竞争效应。该惩罚随模式复杂度提升而加剧(McNemar p < 0.0001),无法仅由提示长度解释。此外,前沿模型并非免疫:在AIME数学竞赛中,Opus 4.7从96.2%降至91.0%(-5.3pp,精确差值为5.26pp)。延迟结构消融实验(先自由推理后格式化)恢复了大部分准确率(3次运行均值:80–87%),支持容量竞争机制。实际启示是:不应回避结构化输出,而是应根据模型容量匹配策略——当模型接近极限时,应先思考再格式化。

原文摘要 · Abstract (English)

Prior work treats structured output as a reasoning tax, but this framing is incomplete: the cost of formatting depends strongly on a model's spare capacity. Using information-matched prose controls and a four-level schema complexity gradient, we separate format-specific effects from prompt-length confounds across 4 models and 5 benchmarks with 0% parse failures on successfully generated responses. We find that structured formats are capacity-dependent. Models with sufficient headroom absorb JSON constraints without degradation (Sonnet: $88.7\pm4.0$% JSON vs. $89.3\pm1.7$% CoT on MATH-Hard). In contrast, formats severely degrade models operating near their limits through two distinct mechanisms. First, under standard token budgets, Haiku drops 36.2pp ($p < 0.0001$) largely due to truncation. Second, even with extended budgets eliminating truncation, GPT-4o-mini drops 28.0pp ($p < 0.001$), revealing pure capacity competition independent of token exhaustion. This format penalty scales with schema complexity (McNemar $p < 0.0001$) and cannot be explained by prompt length alone. Furthermore, these results qualify claims of frontier model immunity: on AIME competition math, Opus 4.7 drops from 96.2% to 91.0% under JSON ($-5.3$pp; the displayed percentages are independently rounded, exact difference is $7/133 = 5.26$pp $\approx 5.3$pp). A delayed-structure ablation -- reasoning freely before formatting -- recovers most of the lost accuracy (3-run mean: 80--87%), supporting the capacity competition mechanism. The practical implication is not to avoid structured output, but to match it to capacity: when a model is near its limits, think first, format later.

结构化推理模型容量格式化损失推理优化

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。