让大模型生成更有结构的长文本，模仿人类写作逻辑。
Align to Structure: Aligning Large Language Models with Structural Information
- 用强化学习结合语言学框架，引导模型生成有层次的文本。
- 在作文和长文档摘要任务中超越标准模型与RLHF优化版本。
- 适合需要高质量长文本生成的研究者与开发者使用。
长文本生成仍是大语言模型的一大挑战，因其缺乏话语生成中的层级规划与结构组织。本文提出结构对齐（Structural Alignment）方法，通过将语言学基础的话语框架融入强化学习，引导模型生成更连贯、结构更清晰的输出。采用近端策略优化框架内的密集奖励机制，基于与人类写作相比的话语独特性，在词元层面分配细粒度奖励。评估了两种互补的奖励模型：第一种通过表面文本特征评分提升可读性并明确结构；第二种通过分析高层话语模式（如层次化话语动机）增强深层连贯性与修辞复杂性，在作文生成和长文档摘要任务中均优于标准模型及经过强化学习人类反馈优化的模型。所有训练数据与代码将在 https://github.com/minnesotanlp/struct_align 公开。
原文摘要 · Abstract (English)
Generating long, coherent text remains a challenge for large language models (LLMs), as they lack hierarchical planning and structured organization in discourse generation. We introduce Structural Alignment, a novel method that aligns LLMs with human-like discourse structures to enhance long-form text generation. By integrating linguistically grounded discourse frameworks into reinforcement learning, our approach guides models to produce coherent and well-organized outputs. We employ a dense reward scheme within a Proximal Policy Optimization framework, assigning fine-grained, token-level rewards based on the discourse distinctiveness relative to human writing. Two complementary reward models are evaluated: the first improves readability by scoring surface-level textual features to provide explicit structuring, while the second reinforces deeper coherence and rhetorical sophistication by analyzing global discourse patterns through hierarchical discourse motifs, outperforming both standard and RLHF-enhanced models in tasks such as essay generation and long-document summarization. All training data and code will be publicly shared at https://github.com/minnesotanlp/struct_align.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。