arXiv:2510.18895cs.SEcs.AI2025-10被引 2

用情绪信号强化代码生成,让模型从尴尬中快速纠错。

CosmoCore Affective Dream-Replay Reinforcement Learning for Code Generation

  • 用轻量MLP给代码生成过程打情感标签,差表现优先重播。
  • 减少48%幻觉代码,自纠错速度提升45%。
  • 适合开发智能编程助手和自动化数据流水线。

我们提出CosmoCore,一种受神经科学启发的强化学习架构,通过融入情感信号来增强大语言模型的代码生成能力。受人类与动物学习机制启发——因犯错感到尴尬而迅速修正——在训练狗狗时一次训斥即可避免重复错误,CosmoCore利用轻量级多层感知机(MLP)为代码生成轨迹标注效价与意外度。高负效价(令人尴尬)的片段,如含错误的代码输出,会被优先放入梦境队列,在离策略更新中重播五倍;而低意外度的成功案例则被剪枝,以防止过度自信和缓冲区溢出。在HumanEval和BigCodeBench等代码生成基准上评估,并结合自定义数据管道环境的仿真测试,CosmoCore使幻觉代码(如语法错误或逻辑缺陷)减少48%,自纠正速度提升45%。在Hugging Face模型与PySpark环境中进行的本地实验验证了这些效果,代码片段可供复现。消融实验表明,效价标注提升了探索中的好奇心,剪枝缓解了效率低下问题。该框架拓展了强化学习从人类反馈(RLHF)到更具情感认知的代码助手机制,适用于集成开发环境(IDE)与数据流水线。代码与自定义微型世界仿真环境已开源。

原文摘要 · Abstract (English)

We introduce CosmoCore, a neuroscience-inspired reinforcement learning (RL) architecture that integrates affective signals to enhance code generation in large language models (LLMs). Motivated by human and animal learning where embarrassment from mistakes drives rapid correction, as observed in training a puppy to avoid repeating errors after a single scolding CosmoCore tags code generation trajectories with valence and surprise using a lightweight multi-layer perceptron (MLP). High-negative valence (cringe) episodes, such as buggy code outputs, are prioritized in a Dream Queue for five-fold replay during off-policy updates, while low-surprise successes are pruned to prevent overconfidence and buffer bloat. Evaluated on code generation benchmarks like HumanEval and BigCodeBench, alongside simulations with a custom data pipeline environment, CosmoCore reduces hallucinated code (e.g., syntax errors or logical bugs) by 48\% and accelerates self-correction by 45\%. Local experiments using Hugging Face models in a PySpark environment validate these gains, with code snippets provided for replication. Ablations confirm valence tagging boosts curiosity in exploration, and pruning mitigates inefficiency. This framework extends RL from human feedback (RLHF) for more emotionally aware code assistants, with applications in IDEs and data pipelines. Code and the custom mini-world simulation are released.

代码生成强化学习情感建模大模型优化

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。