用模拟教学反馈的环境,让数据生成代理自动优化训练数据。
DataEnvGym: Data Generation Agents in Teacher Environments with Student Feedback
- 将数据生成建模为带反馈的序列决策任务,代理自主规划并生成数据。
- 在数学、代码等4个领域中,代理能持续提升学生模型性能。
- 提供多层级结构环境,支持教学策略与反馈机制的测试与对比。
当前训练数据的创建依赖人工分析模型弱点并设计改进数据,虽有大模型作为标注者降低人力,但仍需人工解读评估反馈并引导生成。为实现全自动数据生成代理(教师)的高效测试,我们提出DataEnvGym——一个面向数据生成代理的教师环境测试平台。该平台将数据生成视为包含数据生成策略与引擎的代理,在提供学生反馈的环境中迭代优化。学生在生成数据上训练并评估,其错误或弱项反馈回传给代理以指导下一步生成。DataEnvGym涵盖3种状态表示与动作空间结构层级,支持4个领域(数学、代码、视觉问答、工具使用),可测试多种学生与教师组合。实验表明,示例代理能在不同任务中持续提升学生表现;同时环境可区分不同技能水平,并验证关键模块变体,为未来优化数据生成代理、引擎与反馈机制提供依据。
原文摘要 · Abstract (English)
The process of creating training data to teach models is currently driven by humans, who manually analyze model weaknesses and plan how to create data that improves a student model. Approaches using LLMs as annotators reduce human effort, but still require humans to interpret feedback from evaluations and control the LLM to produce data the student needs. Automating this labor-intensive process by creating autonomous data generation agents - or teachers - is desirable, but requires environments that can simulate the feedback-driven, iterative, closed loop of data creation. To enable rapid, scalable testing for such agents and their modules, we introduce DataEnvGym, a testbed of teacher environments for data generation agents. DataEnvGym frames data generation as a sequential decision-making task, involving an agent consisting of a data generation policy (which generates a plan for creating training data) and a data generation engine (which transforms the plan into data), inside an environment that provides student feedback. The agent's goal is to improve student performance. Students are iteratively trained and evaluated on generated data, and their feedback (in the form of errors or weak skills) is reported to the agent after each iteration. DataEnvGym includes multiple teacher environment instantiations across 3 levels of structure in the state representation and action space. More structured environments are based on inferred skills and offer more interpretability and curriculum control. We support 4 domains (math, code, VQA, and tool-use) and test multiple students and teachers. Example agents in our teaching environments can iteratively improve students across tasks and settings. Moreover, we show that environments teach different skill levels and test variants of key modules, pointing to future work in improving data generation agents, engines, and feedback mechanisms.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。