用小模块对齐语音编码器与大模型，提升口语对话状态追踪效果。
Approaching Dialogue State Tracking via Aligning Speech Encoders and LLMs
- 通过小连接模块对齐语音编码器与大语言模型的表征空间。
- 在SpokenWOZ上达到42.17% JGA，超越当前最佳结果。
- 全开源组件+模糊匹配后处理，适合追求可复现性的研究者。
本文通过一个小连接模块，将语音编码器（WavLM-large）与大语言模型（OLMo、Gemma-2-9B-instruct）的表征空间对齐，实现口语对话状态追踪（DST）。系统采用完全开源组件，实验涵盖全量微调与LoRA适配器微调，并分析了对话历史中代理发言回合的影响及基于模糊匹配的输出后处理。在SpokenWOZ数据集上，最佳模型（WavLM + connector + OLMo-1B）达到34.66% JGA；使用Gemma-2-9B-instruct的系统进一步提升至42.17% JGA。此外，引入Speech-Aware MultiWOZ数据增强训练，显著改善命名实体识别性能。
原文摘要 · Abstract (English)
In this work, we approach spoken Dialogue State Tracking (DST) by bridging the representation spaces of speech encoders and LLMs via a small connector module, with a focus on fully open-sourced and open-data components (WavLM-large, OLMo). We focus on ablating different aspects of such systems including full/LoRA adapter fine-tuning, the effect of agent turns in the dialogue history, as well as fuzzy matching-based output post-processing, which greatly improves performance of our systems on named entities in the dialogue slot values. We conduct our experiments on the SpokenWOZ dataset, and additionally utilize the Speech-Aware MultiWOZ dataset to augment our training data. Ultimately, our best-performing WavLM + connector + OLMo-1B aligned models achieve state of the art on the SpokenWOZ test set (34.66% JGA), and our system with Gemma-2-9B-instruct further surpasses this result, reaching 42.17% JGA on SpokenWOZ test.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。