研究多个大模型联网时的自我演化，发现数据污染会引发模型退化。
LLM Web Dynamics: Tracing Model Collapse in a Network of LLMs
- 构建网络化模型模拟互联网交互，用RAG数据库追踪输出变化。
- 发现多模型协作下输出趋于同质，呈现模型坍塌的收敛模式。
- 理论证明其类比高斯混合模型，为系统性风险提供分析框架。
随着公开互联网生成数据在大语言模型训练中应用增多，数据利用效率提升，但模型坍塌的潜在威胁尚未充分探讨。现有研究多聚焦单一模型或仅依赖统计代理指标。本文提出LLM Web Dynamics（LWD）框架，高效研究网络层面的模型坍塌问题。通过检索增强生成（RAG）数据库模拟互联网环境，分析模型输出的收敛模式。进一步地，基于相互作用高斯混合模型的类比，给出该收敛行为的理论保证。
原文摘要 · Abstract (English)
The increasing use of synthetic data from the public Internet has enhanced data usage efficiency in large language model (LLM) training. However, the potential threat of model collapse remains insufficiently explored. Existing studies primarily examine model collapse in a single model setting or rely solely on statistical surrogates. In this work, we introduce LLM Web Dynamics (LWD), an efficient framework for investigating model collapse at the network level. By simulating the Internet with a retrieval-augmented generation (RAG) database, we analyze the convergence pattern of model outputs. Furthermore, we provide theoretical guarantees for this convergence by drawing an analogy to interacting Gaussian Mixture Models.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。