用麦克风阵列和大模型找回开会时错过的对话内容。
Beamforming-LLM: What, Where and When Did I Miss?
- 通过波束成形分离方向性语音，结合语义检索与生成。
- 支持自然语言提问，精准定位并总结错过的对话片段。
- 适合会议记录、听障辅助及空间感知计算场景。
我们提出Beamforming-LLM系统，使用户能够在多说话人环境中语义召回错过的对话。该系统结合麦克风阵列的空间音频采集与检索增强生成（RAG），支持如“我关注狗的讨论时错过了什么？”等自然语言查询。通过波束成形分离定向音频流，使用Whisper进行转写，并以句向量编码嵌入向量数据库。当接收到用户查询时，系统检索语义相关片段，与未关注段落进行时间对齐，并用轻量级大模型（GPT-4o-mini）生成摘要。最终界面提供对比性摘要、空间上下文和带时间戳的音频回放。本工作为智能听觉记忆系统奠定基础，广泛应用于辅助技术、会议摘要与情境感知个人空间计算。
原文摘要 · Abstract (English)
We present Beamforming-LLM, a system that enables users to semantically recall conversations they may have missed in multi-speaker environments. The system combines spatial audio capture using a microphone array with retrieval-augmented generation (RAG) to support natural language queries such as, "What did I miss when I was following the conversation on dogs?" Directional audio streams are separated using beamforming, transcribed with Whisper, and embedded into a vector database using sentence encoders. Upon receiving a user query, semantically relevant segments are retrieved, temporally aligned with non-attended segments, and summarized using a lightweight large language model (GPT-4o-mini). The result is a user-friendly interface that provides contrastive summaries, spatial context, and timestamped audio playback. This work lays the foundation for intelligent auditory memory systems and has broad applications in assistive technology, meeting summarization, and context-aware personal spatial computing.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。