让大模型生成结构化数据更可靠，只需轻量级强化学习框架
RL-Struct: A Lightweight Reinforcement Learning Framework for Reliable Structured Output in LLMs
- 用分层奖励机制优化策略，无需额外的评判网络
- 生成复杂JSON时结构准确率达89.7%，有效性达92.1%
- 自动形成先学语法后学语义的学习顺序，适合自动化流程场景
概率性大模型生成与确定性结构要求之间的结构鸿沟阻碍了自动化工作流。我们提出RL-Struct，一个基于梯度正则化策略优化（GRPO）的轻量级框架，采用分层奖励函数使大模型满足结构约束。该方法无需评判网络，相比PPO将峰值显存降低38%。在复杂JSON任务上，RL-Struct达到89.7%的结构准确率和92.1%的有效性，显著优于SFT和零样本基线。我们还观察到一种涌现式课程——模型自发优先学习语法结构，再掌握语义内容。模型已公开于https://huggingface.co/Freakz3z/Qwen-JSON。
原文摘要 · Abstract (English)
The Structure Gap between probabilistic LLM generation and deterministic schema requirements hinders automated workflows. We propose RL-Struct, a lightweight framework using Gradient Regularized Policy Optimization (GRPO) with a hierarchical reward function to align LLMs with structural constraints. This approach eliminates the critic network, reducing peak VRAM by 38% compared to PPO. On complex JSON tasks, RL-Struct achieves 89.7% structural accuracy and 92.1% validity, significantly outperforming SFT and zero-shot baselines. We also report an emergent curriculum--a self-organized learning process where the model prioritizes syntax before semantics. Our model is publicly available at https://huggingface.co/Freakz3z/Qwen-JSON.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。