根据任务难度动态选模型,降低企业编码助手的总成本。
Task-to-Model Optimization for Enterprise LLM Coding Assistants: A Data-Driven Framework for Cost-Optimal Routing
- 按任务类型和难度分级,用数据驱动选择最省钱的模型
- 考虑重试、升级和等待时间,总成本比单纯省token更低
- 支持渐进式部署,适合需要降本增效的企业级AI团队
企业级AI编码助手推理成本高昂,仅追求令牌成本最小化往往无法降低端到端成本,因为重试、升级和开发者等待时间都会增加开销。本文提出任务到模型优化(T2MO),一种数据驱动的方法,用于优化生产环境中编码工作流的模型选择。将每个开发者会话视为可发现、分类、分级难度、在类生产环境测试并路由至满足质量与延迟约束的最便宜模型的任务。该框架包含九个阶段:遥测采集、分类体系发现、难度评分、基准构建、候选评估、最优组合推导、支出预测与版本规划、分阶段路由部署及持续治理。不同于以令牌为中心的路由规则,其目标是每完成一个任务的成本,显式纳入失败后升级的代价。我们证明该预期完成成本目标在存在升级场景下弱占优令牌成本最小化,并推导出路由边界——即某模型在特定任务单元中必须达到的最低通过率才能值得部署。决策采用两级层次结构:任务类别难度层级;各单元替代机会聚合为流量加权节省瀑布图,按实际美元收益排序替换候选。该框架支持开发者引导、支出预测,以及从静态策略逐步过渡到影子模式分类器、验证级联,最终实现智能路由。本文详述方法、优化目标、评估协议与治理循环,适用于生产部署与未来实证研究。
原文摘要 · Abstract (English)
Enterprise AI coding assistants incur substantial inference spend, and naive token-cost minimization often fails to reduce end-to-end cost once retries, escalations, and developer wait time are included. We present Task-to-Model Optimization (T2MO), a data-driven methodology for optimizing model selection in production coding workflows. We treat each developer session as a task that can be discovered, classified, graded for difficulty, benchmarked in a production-like harness, and routed to the cheapest model able to complete it within quality and latency constraints. The framework is a nine-stage pipeline spanning telemetry instrumentation, taxonomy discovery, difficulty grading, benchmark construction, candidate evaluation, optimal mix derivation, forecasting and version planning, staged routing deployment, and continuous governance. Unlike token-centric routing rules, our objective is cost per completed task, with failure escalation priced in explicitly. We show that this expected-completion-cost objective weakly dominates token-cost minimization under escalation, and we derive the routing boundary, the minimum pass rate a cheaper model must reach on a given cell to be worth deploying. Decisions are organized as a two-level hierarchy of task category difficulty tier, and per-cell displacement opportunities are aggregated into a traffic-weighted savings waterfall that ranks replacement candidates by realized dollar impact. The framework supports developer guidance, spend forecasting, and a staged transition from static policies to shadow-mode classifiers, verified cascades, and ultimately an intelligent router. We describe the methodology, optimization objective, evaluation protocol, and governance loop in a form suitable for production deployment and future empirical study.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。