让机器人听懂全球语言,自由执行人类指令。
ReLI: A Language-Agnostic Approach to Human-Robot Interaction
- 将大模型转化为跨语言指令到动作的转换器,支持自然对话交互。
- 在140种语言、7万+轮对话上平均准确率达90%以上。
- 适合需要多语言支持的工业、家庭机器人场景。
当前,自主智能体在工业、家庭等日常任务中的应用日益增长,但在全球或跨语言环境下,如何确保与环境有效交互并执行不受语言限制的人类指定任务仍是未解难题。为此,我们提出ReLI,一种语言无关的方法,使自主智能体能够自然对话、对环境进行语义推理,并完成下游任务,无论指令的模态或语言来源。首先,我们将大规模预训练基础模型进行具身化,转化为直接通过自然对话提供常识推理和高层机器人控制的语言-动作模型。其次,通过跨语言适配,确保ReLI在多种语言间具备泛化能力。为验证其鲁棒性,我们在多种短时序和长时序任务上进行了广泛实验,涵盖零样本和少样本空间导航、场景信息检索及查询导向任务。在涉及140种语言、7万+多轮对话的基准测试中,ReLI在跨语言指令解析和任务执行成功率方面平均达到90%±0.2。结果表明,该方法有望推动真实世界中自然人机交互的发展,促进包容性与语言多样性。演示与资源将在https://linusnep.github.io/ReLI/ 公开。
原文摘要 · Abstract (English)
Adapting autonomous agents for real-world industrial, domestic, and other daily tasks is currently gaining momentum. However, in global or cross-lingual application contexts, ensuring effective interaction with the environment and executing unrestricted human-specified tasks regardless of the language remains an unsolved problem. To address this, we propose ReLI, a language-agnostic approach that enables autonomous agents to converse naturally, semantically reason about their environment, and perform downstream tasks, regardless of the task instruction's modality or linguistic origin. First, we ground large-scale pre-trained foundation models and transform them into language-to-action models that can directly provide common-sense reasoning and high-level robot control through natural, free-flow conversational interactions. Further, we perform cross-lingual adaptation of the models to ensure that ReLI generalises across the global languages. To demonstrate ReLI's robustness, we conducted extensive experiments on various short- and long-horizon tasks, including zero- and few-shot spatial navigation, scene information retrieval, and query-oriented tasks. We benchmarked the performance on $140$ languages involving $70K+$ multi-turn conversations. On average, ReLI achieved over $90\%\pm0.2$ accuracy in cross-lingual instruction parsing and task execution success. These results demonstrate its potential to advance natural human-agent interaction in the real world while championing inclusive and linguistic diversity. Demos and resources will be public at: https://linusnep.github.io/ReLI/.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。