CUA-Universe: A Scalable and Dynamic Environment for Hybrid GUI+CLI Agents
CUA-Universe is introduced as a scalable environment-to-data pipeline that turns real desktop software into hybrid GUI+CLI environments. It includes App-Forge, which adapts applications into reproducible VMs and command-line surfaces, scaling to 16 applications; Task-Weave, which synthesizes diverse hybrid tasks of controllable difficulty; and Path-Steer, which steers agent trajectories. The work addresses the scarcity of scalable hybrid environments and the complementary use of GUI and CLI interfaces.
Computer-use agents have advanced on benchmarks like OSWorld and AndroidWorld but still act mostly through the GUI, often producing inefficient trajectories. Real-world computer work is hybrid, combining visual-state inspection with precise, high-throughput command-line operations, so capable agents must coordinate both modalities over shared application state. Yet scalable hybrid environments remain scarce because supporting both GUI and CLI over real applications typically requires substantial manual engineering for each application. Existing agents also struggle to use the two interfaces complementarily: CLI-native agents lack visual perception for tasks involving interface state or layout, while GUI-native agents are inefficient for operations better executed through commands. CUA-Universe is introduced as a scalable environment-to-data pipeline that turns real desktop software into hybrid GUI+CLI environments. App-Forge adapts applications into reproducible VMs and command-line surfaces it discovers, wraps, or generates, scaling to 16 applications; Task-Weave synthesizes diverse hybrid tasks of controllable difficulty from reusable operations over seed files; and Path-Steer steers agent trajectories.
The pipeline addresses the engineering bottleneck of creating hybrid environments by automating application adaptation and task generation. App-Forge's ability to discover, wrap, or generate command-line surfaces suggests a modular approach to exposing CLI functionality alongside GUI. Task-Weave's controllable difficulty and reuse of operations over seed files indicate a systematic method for generating training and evaluation data. Path-Steer likely provides trajectory guidance or correction, potentially improving agent efficiency in hybrid settings.
This work signals a shift toward more realistic and efficient computer-use agents that can leverage both visual and command-line interfaces. The focus on scalability and automation may lower the barrier for developing and benchmarking hybrid agents, potentially accelerating adoption in enterprise automation and developer tools. The emphasis on complementary use of GUI and CLI reflects a practical need for agents that can handle diverse software workflows.
The pipeline could reduce the cost and effort of creating training environments for computer-use agents, enabling faster development of more capable automation products. Enterprises seeking to automate desktop workflows may benefit from agents that efficiently combine GUI and CLI operations. The technology may also support developer tools that require robust interaction with both graphical and command-line interfaces.
Observable next signals include release of the CUA-Universe codebase or benchmark, publication of agent performance comparisons on hybrid tasks, and adoption by other research groups for training or evaluation. Further development may extend the number of supported applications beyond 16 and refine task synthesis for more complex real-world scenarios. Integration with existing agent frameworks could follow.