用哈密顿路径优化训练顺序，让大模型更主动探索。
HAMMER: Hamiltonian Curiosity Augmented Large Language Model Reinforcement
- 用语义多样性构造最小哈密顿路径，动态排序训练样本。
- 在多个基准上提升3%至4%的平均准确率。
- 适合需要增强探索能力的强化学习场景。
当前大语言模型的课程强化学习多依赖难度标注进行数据筛选与排序，但易陷入局部最优，早期过度训练简单样本会削弱模型探索能力。本文提出一种新框架HAMMER，将常用于数据集评估的多样性度量引入动态强化学习过程，通过最小语义哈密顿路径对训练样本排序，使初始训练更具探索性。从泛化界理论视角看，多样性驱动的排序有助于稳定收敛。实验表明，HAMMER能有效激发模型‘好奇心’，在多种推理基准上均实现3%至4%的平均准确率提升。
原文摘要 · Abstract (English)
Recent curriculum reinforcement learning for large language models (LLMs) typically rely on difficulty-based annotations for data filtering and ordering. However, such methods suffer from local optimization, where continual training on simple samples in the early steps can cause the policy to lose its exploration. We propose a novel schema, namely Hamiltonian curiosity augmented large language model reinforcement (HAMMER), that transfers diversity metrics, commonly used in dataset evaluation, into the dynamic reinforcement learning procedure, where training samples are ordered via a minimum-semantic Hamiltonian path making the initial training retrain more exploration. From a theoretical perspective of generalization bounds, diversity-driven ordering facilitates stable convergence. Empirical evaluations indicate that HAMMER stimulates model "curiosity" and consistently achieves a 3% to 4% average accuracy gain across diverse inference benchmark.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。