将文档直接内化到LoRA适配器中，实现无需检索的闭卷问答。
Data Quality over Capacity: Internalizing Documents into LoRA Adapters for Closed-Book QA
- 通过LoRA将文档权重化，直接嵌入模型记忆中。
- 数据质量提升使准确率从57.7%升至85.7%，超过架构改进。
- 适合关注模型内部记忆与训练调试的研究者。
我们研究将文档直接嵌入4位Gemma-4-e4b模型的LoRA权重中，使系统在无检索、无上下文窗口限制的情况下回答关于语料库的问题。在从单文档到99个文档语料库的约100次训练中发现，一旦适配器容量足够，训练数据质量是决定闭卷准确率的主导因素，优于LoRA秩、学习率及两种替代架构的综合影响；容量本身是一个硬性门槛，低于此则任何数据干预均无效。一次数据清洗（将黄金答案缩短为1-6词的标准片段并剔除琐碎信息）使15文档语料库的闭卷准确率从57.7%提升至85.7%，增幅超过所有架构调整。我们验证了容量趋势（秩需随语料规模增长）与秩-学习率耦合关系，初期误判已纠正。在15文档子集上引入真实检索基线：内化适配器（84.2%召回率）优于基于BM25-RAG的基线阅读器（58.9%）甚至现实中的黄金分块基准（65.6%），且延迟更低。报告完整调试过程，含三次误判，作为大模型训练实证调试的案例研究。
原文摘要 · Abstract (English)
We study baking documents directly into the weights of a 4-bit Gemma-4-e4b model via LoRA, so a system can answer questions about a corpus closed-book: no retrieval and no context-window budget. Across roughly 100 training runs from single documents to a 99-document corpus, we find that once adapter capacity is adequate, training-data quality is the dominant lever on closed-book accuracy, outweighing LoRA rank, learning rate, and two alternative architectures combined; capacity itself is a hard gate below which no data intervention helps. A single curation pass (shortening gold answers to canonical 1-6 word spans and dropping trivia) moved closed-book accuracy from 57.7% to 85.7% on a 15-document corpus, a larger jump than any architectural change. We confirm a capacity trend (rank must grow with corpus size) entangled with a coupling between rank and learning rate that we initially misdiagnosed. On a 15-document slice we add a real retrieval baseline: the internalized adapter (84.2% recall) beats a BM25-RAG pipeline with a base reader (58.9%) and even a realistic gold-chunk oracle (65.6%) at lower latency. We report the full arc, including three misdiagnoses, as a case study in debugging LLM training empirically.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。