arXiv:2601.00306cs.CYcs.AI2026-01被引 9

GenAI正悄然摧毁信息可信度,让真假难辨成为常态。

The Generative AI Paradox: GenAI and the Erosion of Trust, the Corrosion of Information Verification, and the Demise of Truth

  • 构建合成现实四层框架:内容、身份、互动与制度
  • 揭示生成式AI导致信任崩塌的六大机制,如成本归零、溯源缺失
  • 案例库涵盖选举欺诈、供应链攻击等真实风险,适合政策制定者参考

生成式AI如今能以极低边际成本大规模生成在感知上令人信服的文本、图像、音频和视频。公众讨论常将其危害归为“深度伪造”或误导信息的增量延伸,但这一视角忽略了更深层的社会技术变革:生成式AI使合成现实成为可能——一种内容、身份与社交互动共同制造、相互强化的连贯、可交互且可能个性化的信息环境。本文认为,最严重的风险并非孤立合成作品的出现,而是共享认知基础与机构验证机制的逐步瓦解,因合成内容、身份与互动变得易生成且难审计。本文(一)将合成现实形式化为内容、身份、互动、制度四层架构;(二)扩展了涵盖个人、经济、信息与社会技术风险的生成式AI危害分类;(三)阐明生成式AI带来的质变:成本崩溃、吞吐量激增、定制化、微细分、溯源空白与信任侵蚀;(四)整合2023–2025年的真实风险事件,形成精炼案例库,展示这些机制如何在欺诈、选举、骚扰、文件伪造与供应链破坏中显现。随后提出缓解策略栈,强调溯源基础设施、平台治理、制度流程重塑与公众韧性应互补而非替代,并提出以测量‘认知安全’为核心的科研议程。最终结论为:生成式媒体泛滥后,社会可能理性地全面否定数字证据。

原文摘要 · Abstract (English)

Generative AI (GenAI) now produces text, images, audio, and video that can be perceptually convincing at scale and at negligible marginal cost. While public debate often frames the associated harms as "deepfakes" or incremental extensions of misinformation and fraud, this view misses a broader socio-technical shift: GenAI enables synthetic realities; coherent, interactive, and potentially personalized information environments in which content, identity, and social interaction are jointly manufactured and mutually reinforcing. We argue that the most consequential risk is not merely the production of isolated synthetic artifacts, but the progressive erosion of shared epistemic ground and institutional verification practices as synthetic content, synthetic identity, and synthetic interaction become easy to generate and hard to audit. This paper (i) formalizes synthetic reality as a layered stack (content, identity, interaction, institutions), (ii) expands a taxonomy of GenAI harms spanning personal, economic, informational, and socio-technical risks, (iii) articulates the qualitative shifts introduced by GenAI (cost collapse, throughput, customization, micro-segmentation, provenance gaps, and trust erosion), and (iv) synthesizes recent risk realizations (2023-2025) into a compact case bank illustrating how these mechanisms manifest in fraud, elections, harassment, documentation, and supply-chain compromise. We then propose a mitigation stack that treats provenance infrastructure, platform governance, institutional workflow redesign, and public resilience as complementary rather than substitutable, and outline a research agenda focused on measuring epistemic security. We conclude with the Generative AI Paradox: as synthetic media becomes ubiquitous, societies may rationally discount digital evidence altogether.

生成式AI信息真实性信任危机社会影响

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。