用压缩重构书本信息，生成更可信的长文本推理数据
CLIPPER: Compression enables long-context synthetic data generation
- 先将书籍压缩为章节摘要，再基于摘要生成复杂命题
- 生成19K条高质量命题，模型准确率从28%提升至76%
- 适合需要长文本推理训练的AI研究者使用
大语言模型开发者越来越依赖合成数据，但生成复杂长文本推理任务的高质量数据仍具挑战。我们提出CLIPPER，一种基于压缩的合成数据生成方法，专用于叙事命题验证——该任务需基于整本书验证一个命题。不同于直接从原始文本生成命题（易产生错误），CLIPPER先将书籍压缩为章节概要和全书摘要，再利用这些中间表示生成复杂命题及其链式思维。相比基线方法，CLIPPER生成的命题更具有效性、依据性和复杂性。我们用CLIPPER构建了一个包含19,000条合成书本命题及其源文本和链式思维的数据集，并用于微调三个开源模型。最优模型在叙事命题验证任务上准确率从28%提升至76%，并在NoCha排行榜上为子100亿参数模型创下新纪录。进一步分析表明，模型生成的链式思维更详细且有据可依，同时在其他叙事理解任务（如NarrativeQA）上也表现更优。
原文摘要 · Abstract (English)
LLM developers are increasingly reliant on synthetic data, but generating high-quality data for complex long-context reasoning tasks remains challenging. We introduce CLIPPER, a compression-based approach for generating synthetic data tailored to narrative claim verification - a task that requires reasoning over a book to verify a given claim. Instead of generating claims directly from the raw text of the book, which results in artifact-riddled claims, CLIPPER first compresses the book into chapter outlines and book summaries and then uses these intermediate representations to generate complex claims and corresponding chain-of-thoughts. Compared to naive approaches, CLIPPER produces claims that are more valid, grounded, and complex. Using CLIPPER, we construct a dataset of 19K synthetic book claims paired with their source texts and chain-of-thought reasoning, and use it to fine-tune three open-weight models. Our best model achieves breakthrough results on narrative claim verification (from 28% to 76% accuracy on our test set) and sets a new state-of-the-art for sub-10B models on the NoCha leaderboard. Further analysis shows that our models generate more detailed and grounded chain-of-thought reasoning while also improving performance on other narrative understanding tasks (e.g., NarrativeQA).
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。