用大模型让视障用户通过对话快速理解手机屏幕内容。
Insight: Enhancing Mobile Accessibility for Blind and Visually Impaired Users with LLMs
- 用大语言模型实现屏幕内容的自然语言实时总结。
- 用户任务时间减少，心理负担降低，更偏好对话界面。
- 适合关注无障碍设计与人机交互创新的研究者。
本文针对现有无障碍服务（如 TalkBack）依赖手动手势和顺序反馈的局限性，提出 Insight——一款基于大语言模型（LLM）的 Android 无障碍服务。该服务支持自然语言交互与屏幕内容实时摘要。通过受控实验对比 Insight 与 TalkBack 的可用性，结果显示：使用 Insight 可显著降低用户心理负担、缩短任务完成时间，且用户更青睐其对话式界面；但用户也反映亟需更好的中断管理机制。研究证明，基于 LLM 的界面能大幅提升移动无障碍体验，并指出融合手势与对话模态的混合方案具有推动包容性设计的潜力。
原文摘要 · Abstract (English)
This research paper addresses the limitations of current mobile accessibility services like TalkBack, which provide manual gesture-based sequential feedback to BVI users. Motivated by the promise of large language models (LLMs), this paper introduces Insight, an Android accessibility service that provides natural language interaction and real-time summarization of the screen. The paper performs a within-subject experimental study with users to compare Insight and TalkBack on usability factors. Results show Insight reduced mental effort and task time, and was preferred because of its dialogue interface, but users felt the need for interruption management. Results show LLM-based interfaces can significantly improve mobile accessibility, and describe the potential of hybrid solutions combining gesture and dialogue modalities towards more inclusive design.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。