用神经网络模拟操作系统界面，输入操作就能生成真实反应。
NeuralOS: Towards Simulating Operating Systems via Neural Generative Models
- 用RNN跟踪状态+扩散模型渲染图像，直接预测屏幕变化。
- 能准确还原鼠标点击、应用启动等交互，生成画面逼真。
- 用合成数据训练可模拟未安装的软件，适合做界面原型设计。
我们提出NeuralOS，一种基于神经生成模型的系统仿真框架，可直接根据鼠标移动、点击和键盘输入等用户操作，预测图形用户界面（GUI）的屏幕帧。该模型结合循环神经网络（RNN）追踪计算机状态与基于扩散的神经渲染器生成屏幕图像。在包含随机交互与由AI代理生成的真实交互的Ubuntu XFCE录制数据集上进行训练。实验表明，NeuralOS能成功生成逼真的GUI序列，准确捕捉鼠标交互，并可靠预测如应用启动等状态转换。除复现现有系统外，该模型还展示出：通过合成训练数据，可让模型模拟从未安装的应用程序（如Doom），并暗示仅从合成演示中学习用户界面的可能性。
原文摘要 · Abstract (English)
We introduce NeuralOS, a neural framework that simulates graphical user interfaces (GUIs) of operating systems by directly predicting screen frames in response to user inputs such as mouse movements, clicks, and keyboard events. NeuralOS combines a recurrent neural network (RNN), which tracks computer state, with a diffusion-based neural renderer that generates screen images. The model is trained on a dataset of Ubuntu XFCE recordings, which include both randomly generated interactions and realistic interactions produced by AI agents. Experiments show that NeuralOS successfully renders realistic GUI sequences, accurately captures mouse interactions, and reliably predicts state transitions like application launches. Beyond reproducing existing systems, NeuralOS shows that synthesized training data can teach the model to simulate applications that were never installed, as illustrated by a Doom application, and suggests a path toward learning user interfaces purely from synthetic demonstrations.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。