Single-screen success is not a real workflow
JarvisGUI, submitted September 9 and accepted to the EMNLP 2026 Main Conference, challenges the way GUI agents are commonly tested. Real work may collect information on a phone, move it into a Windows document and process it with an Ubuntu tool. Intermediate artifacts and shared state must survive each transition, while most benchmarks stay on one device with a fixed script.
The authors represent GUI tasks as input-output transformations under a lightweight type system. This lets them compose multi-step workflows dynamically across virtual Android, Windows and Ubuntu environments and verify whether an agent locates and uses the previous stage's output. The benchmark tests dependency tracking rather than memorization of one click path.
State transfer remains a capability gap
The paper reports that evaluated state-of-the-art open-source GUI agents struggle with cross-device state awareness, contextual reasoning after a platform change and long-horizon dependency management. The result is bounded by its virtual environments and selected agents; it should not be generalized to every commercial system.
For deployment teams, the practical lesson is to measure typed inputs and outputs, file and account boundaries, transition checkpoints and resumability—not only whether the final click occurred.