用推理模型生成领域专用数据，提升复杂文档问答的精准度。
Enhancing Domain-Specific Retrieval-Augmented Generation: Synthetic Data Generation and Evaluation using Reasoning Models
- 用细粒度指标衡量技术文本中信息密度与上下文保留的权衡。
- 小块文本（<10令牌）使精确率提升31%-42%，网络安全需原子分割。
- 基于推理模型生成带断续答案的问答对，适合金融、生物、安全领域。
检索增强生成（RAG）系统在需要从复杂文档中精确提取信息的技术领域表现不佳。现有基于文档级别的评估方法无法捕捉关键的令牌级检索准确率。本文提出一个结合细粒度评估指标与合成数据生成的框架以优化领域特定RAG性能。首先引入面向令牌的指标：精度Ω与交并比（IoU），量化技术文本中上下文保留与信息密度的权衡。其次，设计由指令微调大模型（DeepSeek-R1、DeepSeek-R1蒸馏版、Phi-4）驱动的流水线，为三个专业语料库（SEC 10-K文件、PubMed生物医学摘要、APT威胁报告）生成带有不连续引用跨度的上下文锚定问答对。实证分析显示：小于10令牌的分块可使精度提升31%-42%（IoU=0.071 vs 基线0.053），召回下降18%；领域嵌入策略导致最优分块大小差异达22%（5-20令牌）。DeepSeek-R1-Distill-Qwen-32B在概念对齐上表现最佳（平均IoU高出14%），但无单一配置通用于所有场景。金融文本偏好较大分块（20令牌时召回率达0.81），而网络安全内容适合原子分割（5令牌时精度Ω=0.28）。代码已开源。
原文摘要 · Abstract (English)
Retrieval-Augmented Generation (RAG) systems face significant performance gaps when applied to technical domains requiring precise information extraction from complex documents. Current evaluation methodologies relying on document-level metrics inadequately capture token-resolution retrieval accuracy that is critical for domain-related documents. We propose a framework combining granular evaluation metrics with synthetic data generation to optimize domain-specific RAG performance. First, we introduce token-aware metrics Precision $Ω$ and Intersection-over-Union (IoU) that quantify context preservation versus information density trade-offs inherent in technical texts. Second, we develop a reasoning model-driven pipeline using instruction-tuned LLMs (DeepSeek-R1, DeepSeek-R1 distilled variants, and Phi-4) to generate context-anchored QA pairs with discontinuous reference spans across three specialized corpora: SEC 10-K filings (finance), biomedical abstracts (PubMed), and APT threat reports (cybersecurity). Our empirical analysis reveals critical insights: smaller chunks (less than 10 tokens) improve precision by 31-42% (IoU = 0.071 vs. baseline 0.053) at recall costs (-18%), while domain-specific embedding strategies yield 22% variance in optimal chunk sizing (5-20 tokens). The DeepSeek-R1-Distill-Qwen-32B model demonstrates superior concept alignment (+14% mean IoU over alternatives), though no configuration universally dominates. Financial texts favor larger chunks for risk factor coverage (Recall = 0.81 at size = 20), whereas cybersecurity content benefits from atomic segmentation, Precision $Ω= 0.28$ at size = 5. Our code is available on https://github.com/aryan-jadon/Synthetic-Data-Generation-and-Evaluation-using-Reasoning-Model
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。