让大模型通过声音学习物理常识，理解多普勒效应等真实世界现象。
Teaching Physical Awareness to LLMs through Sounds
- 用物理模拟器生成带真实声源和可控传播路径的训练数据。
- 在模拟与真实场景中实现视线检测、多普勒效应估算等任务。
- 融合声波幅度与相位信息，提升模型对空间物理的理解能力。
大型语言模型（LLMs）在文本和多模态处理上表现卓越，但缺乏对真实世界物理现象的基本认知。本文提出ACORN框架，通过声音教学使LLMs获得物理意识，聚焦多普勒效应、多径效应及空间关系等基本物理现象。为克服数据稀缺问题，ACORN引入基于物理的模拟器，结合真实声源与受控传播通道生成多样化训练数据。基于该模拟器构建了综合性音频问答数据集AQA-PHY，并提出一种能同时处理幅度与相位信息的音频编码器。将此编码器接入先进大模型后，在模拟与真实任务中均取得合理结果，包括视距检测、多普勒效应估计和到达方向估计，为赋予大模型物理世界理解能力开辟新路径。
原文摘要 · Abstract (English)
Large Language Models (LLMs) have shown remarkable capabilities in text and multimodal processing, yet they fundamentally lack physical awareness--understanding of real-world physical phenomena. In this work, we present ACORN, a framework that teaches LLMs physical awareness through sound, focusing on fundamental physical phenomena like the Doppler effect, multipath effect, and spatial relationships. To overcome data scarcity, ACORN introduce a physics-based simulator combining real-world sound sources with controlled physical channels to generate diverse training data. Using this simulator, we build AQA-PHY, a comprehensive Audio Question-Answer dataset, and propose an audio encoder that processes both magnitude and phase information. By connecting our audio encoder to state-of-the-art LLMs, we demonstrate reasonable results in both simulated and real-world tasks, such as line-of-sight detection, Doppler effect estimation, and Direction-of-Arrival estimation, paving the way for enabling LLMs to understand physical world.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。