arXiv:2607.06175cs.CLcs.AI2026-07

用强化学习提升大模型生成流程图质量,关键在奖励函数设计。

Improving LLM-Generated Process Model Quality Through Reinforcement Learning: The Role of Reward Function Design

  • 通过48种奖励组合优化大模型生成流程图,基于38项自动化指标评分。
  • 强化学习使流程图的语法与实用质量显著提升,输出波动降低六倍以上。
  • 等权重奖励优于针对性加权,且不同模型对惩罚项需求差异大。

大型语言模型(LLM)可从自然语言描述生成BPMN流程图,但监督微调(SFT)受限于训练数据模式。强化学习(RL)可通过外部质量度量突破此上限,但多维质量下的奖励函数设计尚未明确。本文系统研究了基于RL的流程图生成中奖励函数设计,使用组序列策略优化(Group Sequence Policy Optimization)在两个模型家族(Llama~3.1 8B、Qwen~2.5 14B)上测试48种配置,奖励来自包含38个指标的自动化评估框架,覆盖语法、语用和语义三方面质量。结果显示:第一,RL显著提升语用与语法质量,同时保持语义一致性,输出变异性减少六倍以上;第二,等权重奖励始终优于定向加权——强调某维度不仅未能改善,反而导致模型陷入低质量状态;第三,设计选择与模型架构存在非线性交互:对一个模型而言无效性惩罚至关重要,对另一个则无关紧要;而SFT初始化对一种架构必不可少,对另一种反而有害。这些发现表明,奖励构成是优化结果的关键决定因素,其影响可达是否采用强化学习本身。结论可推广至所有需多维度自动评估的结构化生成任务。代码与实现已开源:https://github.com/chlauer99/RL_for_process_modeling。

原文摘要 · Abstract (English)

Large language models (LLMs) can generate BPMN process models from natural-language descriptions, yet supervised fine-tuning (SFT) limits their output quality to the patterns present in the training data. Reinforcement learning (RL) can optimize beyond this ceiling using external quality measures, but how the reward function should be designed when quality is multi-dimensional remains unexplored. We present a systematic investigation of reward function design for RL-based process model generation, training two LLM families (Llama~3.1 8B, Qwen~2.5 14B) under 48 configurations using Group Sequence Policy Optimization with rewards derived from an automated evaluation framework comprising 38 metrics across syntactic, pragmatic, and semantic quality. Three findings emerge. First, RL significantly improves pragmatic and syntactic quality while preserving semantic fidelity, reducing output variability by more than sixfold. Second, equal reward weighting consistently outperforms targeted weighting: emphasizing a specific dimension fails to improve it and can collapse the model into a low-quality mode. Third, design choices interact with model architecture in non-trivial ways: the invalidity penalty is essential for one model but irrelevant for the other, and SFT initialization is indispensable for one architecture but counterproductive for another. These results demonstrate that reward composition is a primary determinant of optimization outcomes, with effects as large as the decision to apply RL itself. The findings generalize to any structured generation task where quality is assessed along multiple automated dimensions. We release our implementation and experimental code at https://github.com/chlauer99/RL_for_process_modeling.

强化学习流程建模大模型质量评估

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。