JarvisGUI: Towards Cross-Device GUI Agents with Dynamic Task Composition
JarvisGUI is a dynamic benchmark that evaluates GUI agents on cross-device workflows requiring coordinated interaction across Android, Windows, and Ubuntu. It formulates GUI tasks as input-output transformations under a lightweight type system, enabling automatic composition of multi-step, cross-device workflows and dynamic evaluation within a unified framework. Evaluation in virtual environments spanning multiple operating systems reveals that state-of-the-art open-source GUI agents struggle with state-transfer awareness, cross-platform contextual reasoning, and long-horizon dependency management.
JarvisGUI introduces a benchmark for cross-device GUI agents, addressing the gap left by existing single-device, statically defined benchmarks. It uses a type system to compose workflows across Android, Windows, and Ubuntu, and evaluates agents in virtual environments. The benchmark shows that current open-source GUI agents have significant weaknesses in state-transfer awareness, cross-platform contextual reasoning, and long-horizon dependency management.
The benchmark's lightweight type system enables automatic composition of cross-device tasks, providing a unified evaluation framework. The observed failures in state-transfer awareness and long-horizon dependency management suggest that current agent architectures lack robust mechanisms for maintaining shared state and context across heterogeneous platforms.
The findings indicate that GUI agents are not yet ready for real-world cross-device workflows, which may slow enterprise adoption of agent-based automation across multiple operating systems. This creates an opportunity for improved agent frameworks that handle cross-platform state and context.
For businesses, the benchmark highlights a critical gap in current GUI agent capabilities, suggesting that investment in cross-device agent solutions could yield competitive advantage in automation of multi-platform workflows.
Next signals to watch include new agent architectures or training methods specifically targeting cross-device state management, and follow-up benchmarks that extend JarvisGUI with more complex workflows or additional platforms.