arXiv:2606.04362cs.IRcs.CL2026-06被引 1

通过对照实验发现,ChatGPT引流效果被平台增长夸大了。

Disentangling Answer Engine Optimization from Platform Growth: A Log-Based Natural Experiment on ChatGPT Referral Traffic

  • 用同一网站未干预页面作对照,分离出真实AEO效果
  • 干预后流量提升1.82倍,但统计显著性较弱
  • 提醒读者警惕夸大其词的“引流增长”宣传

大型语言模型(如ChatGPT)已成为向开放网络导流的重要源头,催生了类似搜索引擎优化的Answer Engine Optimization(AEO)。公开案例常宣称流量增长数倍,但这些数据被答案引擎自身的平台级增长所混淆。本研究对高流量域名glasp.co开展纵向实地实验,其数十万条YouTube问答页面在2026年1月接受了明确的AEO干预。由于干预集中于部分页面,其余未干预页面作为同期对照,吸收平台整体增长带来的“顺风”。基于第一方分析与服务器日志,而非第三方估算:(1)原始流量增长受平台增长主导——总聊天机器人引流量月均增长5.7倍,未干预页面也增长3.5倍;(2)以每周干预/对照比值构建中断时间序列模型,估计干预带来1.82倍的水平提升(95%置信区间1.31–2.54,HAC p=0.001),在剔除低参与度流量后仍达2.27倍,且在多种设定下稳健;(3)然而,保守的时间类比置换检验结果为p=0.16,表明效应尚不充分确证,因前期数据短且嘈杂;(4)谷歌有机点击未出现异常下降,索引状态保持正常,符合SEO保护规则。方法论启示:使用站内对照分离处理效应与平台趋势,远比单一增长倍数重要,提示主流AEO倍数可能严重夸大因果影响。

原文摘要 · Abstract (English)

Large language model (LLM) "answer engines" such as ChatGPT now send measurable referral traffic to the open web, and a practice analogous to search engine optimization, here called Answer Engine Optimization (AEO), has emerged. Public AEO success stories typically quote large raw growth multiples, but raw referral growth is confounded by the rapid platform-level growth of the answer engines themselves. We report a longitudinal field study on a single high-traffic domain (glasp.co) whose corpus of hundreds of thousands of YouTube question-and-answer pages received a defined bundle of AEO interventions in January 2026 (detailed in Section 4). Because the interventions were concentrated on one subset of the site, the untreated remainder of the same domain acts as a contemporaneous control that absorbs the platform tailwind. Using first-party analytics and server logs rather than probabilistic third-party estimators, we find: (1) raw growth is dominated by the platform tailwind: on monthly aggregates total ChatGPT referrals grew 5.7x while untreated pages on the same domain grew 3.5x over the same window; (2) an interrupted time-series model on the weekly treated/control ratio estimates a discrete, intervention-aligned level increase of 1.82x (95% CI 1.31-2.54, HAC p=0.001), robust across engagement-filtered traffic (2.27x) and alternative specifications; (3) however, a conservative placebo-in-time permutation test yields p=0.16, so the effect is suggestive, not conclusive, given a short and noisy pre-period; and (4) Google organic clicks to treated pages did not fall beyond the ambient site-wide trend and indexation was preserved, consistent with the SEO-protection rule. The methodological message, separating treatment from platform tailwind with an on-domain control, matters more than any single multiple, and implies that headline AEO multiples substantially overstate causal effect.

AEO流量分析因果推断

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。