通过组合搜索提升对话状态追踪的上下文示例选择效果
Improving Dialogue State Tracking through Combinatorial Search for In-Context Examples
- 基于示例组合效应优化检索评分，提升上下文示例质量
- 在MultiWOZ上实现20倍数据效率提升，SGD上表现良好
- 适合追求高精度对话系统与数据高效训练的研究者
在对话状态追踪（DST）中，上下文学习包含一个检索器，用于选取带标签的对话作为上下文示例，以及一个DST模型，利用这些示例推断查询对话的状态。现有构建检索器训练数据的方法存在三大局限：（1）未考虑示例间的协同效应，（2）未充分考虑查询的语言特征，（3）评分未直接针对DST性能优化。因此，检索器可能无法选出显著提升DST性能的示例。为此，我们提出CombiSearch，一种基于示例组合对DST性能影响进行评分的方法。在MultiWOZ上的评估表明，使用CombiSearch训练的检索器超越当前最优模型，在无检索错误假设下，上界DST性能相比传统方法提升12%绝对值。同时，实现了20倍的数据效率增益，并在SGD数据集上表现出良好泛化能力。该结果显著提升了实际DST性能的上限，表明现有方法依赖次优数据训练检索器。
原文摘要 · Abstract (English)
In dialogue state tracking (DST), in-context learning comprises a retriever that selects labeled dialogues as in-context examples and a DST model that uses these examples to infer the dialogue state of the query dialogue. Existing methods for constructing training data for retrievers suffer from three key limitations: (1) the synergistic effect of examples is not considered, (2) the linguistic characteristics of the query are not sufficiently factored in, and (3) scoring is not directly optimized for DST performance. Consequently, the retriever can fail to retrieve examples that would substantially improve DST performance. To address these issues, we present CombiSearch, a method that scores effective in-context examples based on their combinatorial impact on DST performance. Our evaluation on MultiWOZ shows that retrievers trained with CombiSearch surpass state-of-the-art models, achieving a 20x gain in data efficiency and generalizing well to the SGD dataset. Moreover, CombiSearch attains a 12% absolute improvement in the upper bound DST performance over traditional approaches when no retrieval errors are assumed. This significantly increases the headroom for practical DST performance while demonstrating that existing methods rely on suboptimal data for retriever training.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。