让训练模型与部署环境一致，提升代码生成的准确性和可靠性。
Train What You Deploy:Token-Faithful Post-Training of a Production Coding
- 保留原始提示采样，避免训练与部署环境不匹配
- 在相同条件下，性能比标准方法提升3.0分
- 适合对生成准确性要求高的生产级代码系统
现有代码与终端代理的后训练流程存在严重的令牌和控制保真度误差：简化训练环境与生产部署不匹配，且从代理日志离线重建令牌会扭曲原始提示，并混淆策略调用与模型后台操作。我们提出一种保真度感知的训练耦合框架，保留训练端对原始提示的采样，通过协商式训练协议消除虚假模型调用，并将损失计算限制在可验证的令牌片段上，具备闭合失败保证。进一步提出认证分歧近似策略优化（C-DPPO），在标准DPPO基础上建立紧密的双向总变差认证边界、自适应K规则、预算感知序列保证及抗错误策略掩码。在匹配的Baize5B和Baize10B模型上，采用相同的训练与测试协议在TMax-100数据集上评估，C-DPPO相较标准DPPO在不同模型规模下均实现+3.0点性能提升。证书审计验证了训练管道的可靠性和完整运行覆盖。
原文摘要 · Abstract (English)
Existing post-training pipelines for coding and terminal agents suffer severe token and control fidelity errors: simplified training environments mismatch production deployments, and offline token reconstruction from agent logs distorts original prompts and conflates policy calls with background model operations. We present a fidelity-aware training coupling framework that retains trainer-side sampling over original prompts, eliminates spurious model calls via a negotiated training protocol, and restricts loss computation to verifiable token spans with closed-failure guarantees. We further propose Certified Divergence Proximal Policy Optimization (C-DPPO), which establishes tight two-sided TV certification bounds, adaptive-K rules, budget-aware sequence guarantees, and error-robust policy masking atop standard DPPO. Evaluated on matched Baize5B and Baize10B models with identical training and test protocols on TMax-100, C-DPPO yields a consistent +3.0-point performance gain over standard DPPO across model scales. Certificate audits validate the reliability and full operational coverage of our certified training pipeline.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。