MATE用多智能体系统为残障用户自动转换信息模态,提升数字可及性。
MATE: LLM-Powered Multi-Agent Translation Environment for Accessibility Applications
- 基于用户需求动态转换图文等信息模态,如图像转语音
- 自研任务识别模型在定制数据集上优于主流LLM和统计模型
- 支持本地部署与机构系统集成,保障隐私且适配多种硬件
无障碍性仍是当今社会的关键挑战,许多技术未能充分满足用户需求。现有多智能体系统(MAS)因闭源设计缺乏定制化,难以全面支持有需求的用户,导致残障人士在交互数字环境时面临重大障碍。我们提出MATE,一种多模态无障碍多智能体系统,可根据用户需求执行模态转换。例如,对视障用户,系统将图像转换为音频描述。该系统适用于医疗等多个领域,可作为各类用户的实用助手。支持从LLM API调用到自定义机器学习分类器的多种模型,具备广泛硬件兼容性。系统可本地运行,保障敏感信息隐私安全,并能有效集成至机构技术(如数字医疗服务)中实现实时辅助。此外,我们提出ModCon-Task-Identifier模型,可精准提取用户输入中的模态转换任务。大量实验表明,该模型在自建数据集上持续优于其他LLM和统计模型。代码与数据已公开于https://github.com/AlgazinovAleksandr/Multi-Agent-MATE。
原文摘要 · Abstract (English)
Accessibility remains a critical concern in today's society, as many technologies are not developed to support the full range of user needs. Existing multi-agent systems (MAS) often cannot provide comprehensive assistance for users in need due to the lack of customization stemming from closed-source designs. Consequently, individuals with disabilities frequently encounter significant barriers when attempting to interact with digital environments. We introduce MATE, a multimodal accessibility MAS, which performs the modality conversions based on the user's needs. The system is useful for assisting people with disabilities by ensuring that data will be converted to an understandable format. For instance, if the user cannot see well and receives an image, the system converts this image to its audio description. MATE can be applied to a wide range of domains, industries, and areas, such as healthcare, and can become a useful assistant for various groups of users. The system supports multiple types of models, ranging from LLM API calling to using custom machine learning (ML) classifiers. This flexibility ensures that the system can be adapted to various needs and is compatible with a wide variety of hardware. Since the system is expected to run locally, it ensures the privacy and security of sensitive information. In addition, the framework can be effectively integrated with institutional technologies (e.g., digital healthcare service) for real-time user assistance. Furthermore, we introduce ModCon-Task-Identifier, a model that is capable of extracting the precise modality conversion task from the user input. Numerous experiments show that ModCon-Task-Identifier consistently outperforms other LLMs and statistical models on our custom data. Our code and data are publicly available at https://github.com/AlgazinovAleksandr/Multi-Agent-MATE.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。