用价值加权路由优化推荐系统,降低误判成本。
How Often Should a Recommender Call an LLM? Value-Weighted Routing, Monitoring, and Seasonal Robustness
- 根据物品价值与难度双重判断,决定是否调用昂贵大模型
- 精度达98.3%,比仅看难度的方案高4个百分点
- 适合需要精准控制成本的电商推荐场景
推荐系统在廉价启发式方法与昂贵大语言模型(LLM)之间做路由决策时,通常以难度为标准:将难题交给大模型。我们指出这一思路不完整,因为难度和商业价值是不同维度——难但便宜的物品与难且贵的物品,其错误代价并不相同。为此提出Value Router,一个完全基于合成数据的零售商品管道模拟系统,仅依赖对难度和价值的估计进行路由,不使用真实标签。研究分三阶段:第一,对比价值加权阈值路由、仅难度路由和随机基线,在类别销量与价值呈反相关性的合成目录中,价值加权方案召回率与难度基线一致(60%),但精度提升至98.3%(优于94.3%);第二,通过决策日志与监控发现,聚合指标掩盖了真正的失败模式,结果主要由类别间差异驱动而非个体区分;第三,模拟黑五需求激增(2.5倍量级,高价值品类占比上升),比较静态路由、季节性调优路由及两种慢路径预算策略。所有结果均来自可控合成模拟,具有明确人工真实标签,旨在揭示成本感知路由系统的设计原则,而非验证现实世界效果。
原文摘要 · Abstract (English)
Routing decisions between a cheap heuristic and an expensive large language model (LLM) are typically framed as a difficulty problem: send the hard cases to the expensive path. We argue this framing is incomplete because difficulty and business value are distinct axes - a difficult cheap item and a difficult costly item do not have the same cost of error. We present Value Router, a fully synthetic simulation of a retail merchandising pipeline that routes items using only estimated difficulty and estimated value, never ground truth. The study has three stages. First, a value-weighted threshold router is compared with a difficulty-only and a random baseline on a synthetic catalog with an inverse correlation between category volume and value. Value-weighting matches the difficulty-only baseline's recall of true high-value items (60%) while achieving substantially higher precision (98.3% vs. 94.3%). Second, a decision logger and monitor expose a failure mode hidden by aggregate metrics showing that the aggregate result is driven almost entirely by between-category differences rather than per-item discrimination. Third, a simulated Black Friday demand surge (2.5 volume with a shift toward higher-value categories) compares a static router, a seasonally tuned router, and two slow-path budget policies. All results are from a controlled synthetic simulation with experimenter-defined ground truth and illustrate design principles for cost-aware routing systems rather than validated real-world claims.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。