用强化学习提升大模型生成合规JSON的能力
Learning to Generate Structured Output with Schema Reinforcement Learning
- 引入细粒度校验器，通过强化学习训练模型理解JSON结构
- 在4万多个JSON模式上测试，最新模型仍难以生成有效输出
- 显著提升模型生成准确率及下游任务表现，适合接口集成场景
本研究探讨大语言模型（LLM）生成结构化输出的能力，重点是根据给定模式生成有效的JSON。尽管JSON广泛用于语言模型与程序的集成，但对其生成能力缺乏系统分析与基准测试。我们从结构理解、转义处理和自然语言描述等角度，探究如何评估并提升模型生成有效响应的能力。基于此，我们构建了包含约4万种不同JSON模式的SchemaBench，用于评估模型生成有效JSON的能力。结果发现，当前最先进的大模型仍难以生成有效JSON字符串。此外，我们证明将强化学习与细粒度模式校验器结合，可进一步增强模型对JSON模式的理解，从而显著提升生成性能和下游任务表现。
原文摘要 · Abstract (English)
This study investigates the structured generation capabilities of large language models (LLMs), focusing on producing valid JSON outputs against a given schema. Despite the widespread use of JSON in integrating language models with programs, there is a lack of comprehensive analysis and benchmarking of these capabilities. We explore various aspects of JSON generation, such as structure understanding, escaping, and natural language description, to determine how to assess and enable LLMs to generate valid responses. Building upon this, we propose SchemaBench features around 40K different JSON schemas to obtain and assess models' abilities in generating valid JSON. We find that the latest LLMs are still struggling to generate a valid JSON string. Moreover, we demonstrate that incorporating reinforcement learning with a Fine-grained Schema Validator can further enhance models' understanding of JSON schema, leading to improved performance. Our models demonstrate significant improvement in both generating JSON outputs and downstream tasks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。