让大模型学会识别和利用对话中的引用内容,提升问答准确性。
Mind the Quote: Enabling Quotation-Aware Dialogue in LLMs via Plug-and-Play Modules
- 将引用定位与生成解耦,构建基于引用跨度的对话生成框架。
- 在五种场景下实现90%以上正确率,支持未见主题泛化。
- 轻量模块可即插即用,仅更新2.8%参数,推理无额外开销。
人类与AI对话常依赖引用前文内容,但现有大模型缺乏显式引用定位与利用机制。本文将该问题形式化为跨度条件生成,将每轮对话分解为对话历史、一组标记偏移的引用跨度和意图语句。基于此抽象,提出面向引用的自动化数据流水线,可合成特定任务对话,通过多阶段一致性验证确保答案正确性,生成首个涵盖五种典型场景的基准数据集。为满足零开销与参数高效要求,提出QuAda方法:在每个注意力头附加两个瓶颈投影,推理时动态增强或抑制引用跨度注意力,保持提示不变且仅更新<2.8%主干权重。实验表明,QuAda适用于所有场景且能泛化至未见主题,是高效的即插即用引用感知对话解决方案。
原文摘要 · Abstract (English)
Human-AI conversation frequently relies on quoting earlier text-"check it with the formula I just highlighted"-yet today's large language models (LLMs) lack an explicit mechanism for locating and exploiting such spans. We formalise the challenge as span-conditioned generation, decomposing each turn into the dialogue history, a set of token-offset quotation spans, and an intent utterance. Building on this abstraction, we introduce a quotation-centric data pipeline that automatically synthesises task-specific dialogues, verifies answer correctness through multi-stage consistency checks, and yields both a heterogeneous training corpus and the first benchmark covering five representative scenarios. To meet the benchmark's zero-overhead and parameter-efficiency requirements, we propose QuAda, a lightweight training-based method that attaches two bottleneck projections to every attention head, dynamically amplifying or suppressing attention to quoted spans at inference time while leaving the prompt unchanged and updating < 2.8% of backbone weights. Experiments across models show that QuAda is suitable for all scenarios and generalises to unseen topics, offering an effective, plug-and-play solution for quotation-aware dialogue.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。