用好奇心驱动的强化学习，让小模型写出接近大模型的精彩故事。
COS(M+O)S: Curiosity and RL-Enhanced MCTS for Exploring Story Space via Language Models
- 结合蒙特卡洛树搜索与好奇度奖励，引导小模型探索多样剧情分支。
- 在短篇故事任务中，生成内容质量接近70B大模型，胜过3B基线0.59标准差。
- 适合对剧情生成、强化学习应用感兴趣的开发者和研究者。
我们提出COS(M+O)S，一种受系统2启发的开放式剧情生成框架，能系统探索庞大的故事扩展空间。该方法通过蒙特卡洛树搜索（MCTS），结合基于步骤的值模型——该模型奖励适度意外性（好奇度）并惩罚逻辑不连贯性，并利用几率比偏好优化（ORPO）对策略进行微调，以提升高价值剧情分支的表现。这一迭代强化学习循环可系统探索多个候选剧情分支，回传质量信号并优化策略，显著推动策略从基于解谜的思维链转向更具人物驱动的故事叙述。在小型短篇故事测试中，67%-77%参与者更偏好COS(M+O)S最高评分的剧情，表明其学习到的值函数具有有效性。GPT-4o评估显示，该方法相较Llama 3.2 3B的单次解码高出0.59标准差，仅落后于Llama 3.1 70B 0.06标准差（无显著差异，p=0.93）。与o1对比，其优于3B基线1.5标准差，且与70B模型无统计差异。然而，绝对故事质量仍受限于小模型容量和有限训练数据。
原文摘要 · Abstract (English)
We present COS(M+O)S, a System 2-inspired framework for open-ended plot development that systematically explores the vast space of possible story expansions, enabling a 3B-parameter language model to approach the plot quality of a 70B model on select short-story tasks. The method accomplishes this by combining Monte Carlo Tree Search (MCTS), guided by a step-level value model that rewards moderate surprisal (curiosity) while penalizing incoherence, and Odds Ratio Preference Optimization (ORPO) to fine-tune the policy on high-value plot expansions. This iterative reinforcement learning loop systematically explores multiple candidate plot branches, backpropagates quality signals, and adapts the policy for faster convergence, notably shifting the policy from puzzle-based Chain-of-Thought to more character-driven storytelling. In small-scale tests with short-story prompts, 67%-77% of participants favored COS(M+O)S's highest-rated expansions over lower-rated ones, suggesting that our learned value function aligns. GPT-4o ratings further show that COS(M+O)S surpasses naive single-pass decoding from Llama 3.2 3B by 0.59 SD, coming within 0.06 SD of Llama 3.1 70B (no significant difference, p=0.93). Pairwise comparisons with o1 place COS(M+O)S 1.5 SD above the 3B baseline and find no statistically significant gap from 70B. Nevertheless, absolute story quality remains modest, constrained by the small model's capacity and limited training data.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。