可实时追溯大模型输出与万亿级训练数据的原文匹配。
OLMoTrace: Tracing Language Model Outputs Back to Trillions of Training Tokens
- 基于扩展版infini-gram，实现输出与训练文本的快速比对。
- 数秒内返回结果，支持验证事实、识别幻觉和分析创意来源。
- 开源免费，适合研究者验证模型行为与数据关联性。
我们提出OLMoTrace，首个能在实时中将语言模型输出追溯至完整多万亿级训练数据的系统。该系统通过查找模型输出片段与训练语料中文档的原文匹配，揭示模型生成内容的数据来源。基于Liu等（2024）提出的扩展版infini-gram技术，系统可在数秒内返回追溯结果。OLMoTrace有助于用户从训练数据视角理解模型行为，可应用于事实核查、幻觉检测及语言模型创造力分析。系统已公开并完全开源。
原文摘要 · Abstract (English)
We present OLMoTrace, the first system that traces the outputs of language models back to their full, multi-trillion-token training data in real time. OLMoTrace finds and shows verbatim matches between segments of language model output and documents in the training text corpora. Powered by an extended version of infini-gram (Liu et al., 2024), our system returns tracing results within a few seconds. OLMoTrace can help users understand the behavior of language models through the lens of their training data. We showcase how it can be used to explore fact checking, hallucination, and the creativity of language models. OLMoTrace is publicly available and fully open-source.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。