用强化学习让大模型生成严格符合格式的JSON数据。
Think Inside the JSON: Reinforcement Strategy for Strict LLM Schema Adherence
- 通过合成推理数据+定制奖励函数训练小模型的结构化思维能力。
- 仅用20小时在8张H100上训练，就能稳定输出合规JSON。
- 适合需要精准数据格式的应用，如医疗、金融系统对接。
本文针对大语言模型生成时难以严格遵守数据模式的问题，提出基于强化学习的思考策略（ThinkJSON）。该方法在1.5B参数模型上，利用20K条无结构到结构的合成数据进行深度求索R1式强化学习，建立核心推理能力；再在另10K条推理样本上进行监督微调，聚焦于下游任务中的模式一致性。尽管训练规模较小，仅需约20小时在8xH100 GPU集群完成GRPO训练，3小时在1xA100完成SFT，模型仍表现出色。在与原版DeepSeek R1（671B）、Qwen-1.5B/7B及Gemini 2.0 Flash（70B）的对比中，本方法展现出在真实场景下的高效与可靠性，验证了资源节省型框架在模式约束生成中的实用性。
原文摘要 · Abstract (English)
In this paper, we address the challenge of enforcing strict schema adherence in large language model (LLM) generation by leveraging LLM reasoning capabilities. Building on the DeepSeek R1 reinforcement learning framework, our approach trains structured reasoning skills of a 1.5B parameter model through a novel pipeline that combines synthetic reasoning dataset construction with custom reward functions under Group Relative Policy Optimization (GRPO). Specifically, we first perform R1 reinforcement learning on a 20K sample unstructured-to-structured dataset, mirroring the original DeepSeek R1 methods, to establish core reasoning abilities. Subsequently, we performed supervised fine-tuning on a separate 10K reasoning sample dataset, focusing on refining schema adherence for downstream tasks. Despite the relatively modest training scope, requiring approximately 20 hours on an 8xH100 GPU cluster for GRPO training and 3 hours on 1xA100 for SFT, our model demonstrates robust performance in enforcing schema consistency. We compare our ThinkJSON approach against the original DeepSeek R1 (671B), distilled versions of DeepSeek R1 (Qwen-1.5B and Qwen-7B), and Gemini 2.0 Flash (70B), showcasing its effectiveness in real-world applications. Our results underscore the practical utility of a resource-efficient framework for schema-constrained text generation.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。