用眼神控制生成和编辑图像视频，实时精准追踪眼球位置。
GazeGen: Gaze-Driven User Interaction for Visual Content Generation
- 通过轻量级眼动模型实现用户眼动实时预测，支持边缘设备运行。
- 在树莓派4上实现低延迟、高精度眼动追踪，误差小且响应快。
- 支持眼神控制图像增删、移动、变风格及静态图转视频，交互自然。
我们提出 GazeGen，一个基于用户眼动生成和操控视觉内容（图像与视频）的系统。用户只需注视目标区域，即可通过眼动完成图像添加/删除、重定位、表面风格变化，以及将静态图像转换为视频。核心是仅281K参数的DFT Gaze（蒸馏与微调眼动）模型，通过创新的知识蒸馏与适配器微调技术，从10倍大的模型中压缩而来，在小型边缘设备上实现个性化、实时的眼动预测。该系统首次将视觉内容生成与实时眼动估计结合，输入为用户眼图像和视场，输出为眼动点与生成内容。在AEA与OpenEDS2020基准测试中，模型展现出低角度误差与低延迟表现，验证了其高效性。应用场景展示其在多种场景下的实用性与灵活性。
原文摘要 · Abstract (English)
We present GazeGen, a user interaction system that generates visual content (images and videos) for locations indicated by the user's eye gaze. GazeGen allows intuitive manipulation of visual content by targeting regions of interest with gaze. Using advanced techniques in object detection and generative AI, GazeGen performs gaze-controlled image adding/deleting, repositioning, and surface style changes of image objects, and converts static images into videos. Central to GazeGen is the DFT Gaze (Distilled and Fine-Tuned Gaze) agent, an ultra-lightweight model with only 281K parameters, performing accurate real-time gaze predictions tailored to individual users' eyes on small edge devices. GazeGen is the first system to combine visual content generation with real-time gaze estimation, made possible exclusively by DFT Gaze. This real-time gaze estimation enables various visual content generation tasks, all controlled by the user's gaze. The input for DFT Gaze is the user's eye images, while the inputs for visual content generation are the user's view and the predicted gaze point from DFT Gaze. To achieve efficient gaze predictions, we derive the small model from a large model (10x larger) via novel knowledge distillation and personal adaptation techniques. We integrate knowledge distillation with a masked autoencoder, developing a compact yet powerful gaze estimation model. This model is further fine-tuned with Adapters, enabling highly accurate and personalized gaze predictions with minimal user input. DFT Gaze ensures low-latency and precise gaze tracking, supporting a wide range of gaze-driven tasks. We validate the performance of DFT Gaze on AEA and OpenEDS2020 benchmarks, demonstrating low angular gaze error and low latency on the edge device (Raspberry Pi 4). Furthermore, we describe applications of GazeGen, illustrating its versatility and effectiveness in various usage scenarios.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。