不同AI推荐系统分歧大,但失败原因高度一致。
Divergent Recommendations, Convergent Diagnoses: Cross-Provider Failure-Mode Convergence in AI Commercial Recommendation
- 通过3个失败模式诊断,发现两模型对推荐失败原因判断高度一致。
- 长尾品牌失败诊断一致率达99.6%,领先品牌为81%。
- 修复诊断出的问题能同时提升两平台可见性,定位优化更适配单平台。
一家使用ChatGPT和Claude进行产品推荐的品牌面临策略选择:采用统一优化方案,还是针对不同提供商分别调整?在四个批次共215个商业场景提示下,两模型推荐结果的重合度仅为Jaccard 0.35(低于同提示重跑基准0.50–0.61)。推荐结果分歧明显,但当两者均未推荐某品牌时,按三种失败模式分类——可发现性(品牌未进入模型)、吸引力(进入但未提及)、定位性(提及但未推荐)——在7,763次共同失败中,两模型诊断一致率达95.1%(95%置信区间[94.3%, 95.7%])。一致性随品牌知名度下降而上升:品类龙头为81% [78.2%, 84.0%],长尾区域性品牌达99.6% [99.3%, 99.9%]。两模型生成路径差异显著:Anthropic基于先验推荐占比43–52%,OpenAI为8–29%。然而在长尾场景下,它们对失败根源的判断趋于一致。修复诊断出的问题可同步提升双平台可见性;而针对品类领导者的定位与内容优化则更具平台特异性。
原文摘要 · Abstract (English)
A brand whose customers use both ChatGPT and Claude for product recommendations faces a strategic choice: a single optimization playbook, or one per provider? Across 215 commercially-framed prompts in four measurement batches, the two providers disagree on which brands they recommend roughly two-thirds of the time (cross-provider recommendation Jaccard 0.35, below the 0.50-0.61 same-prompt rerun baseline). The picks diverge. But when neither provider recommends a brand, we classify the failure into one of three modes -- discoverability (the brand never reaches the model), compellingness (it reaches the model but isn't mentioned), or positioning (it's mentioned but not recommended) -- and on 7,763 such joint failures, both providers diagnose the same failure mode 95.1% of the time (clustered 95% CI [94.3%, 95.7%]). Agreement rises monotonically with falling brand prominence, from 81% [78.2%, 84.0%] on category leaders to 99.6% [99.3%, 99.9%] on long-tail regional brands. The two providers reach their picks by measurably different generative routes -- Anthropic recommends from priors 43-52% of the time, OpenAI 8-29% -- but they converge on the failure diagnosis where it matters most for the long tail. Work that addresses the diagnosed failure mode lifts visibility on both providers; positioning - and content-level work for category leaders is more provider-specific.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。