Show-Harness: Just a VLM Agent Can Play Robots
Show-Harness is an Embodied Harness that enables vision-language models (VLMs) to control robots through a compact semantic interface. It exposes discrete semantic action units that VLMs can reason over, while embodiment-specific interpreters ground them into local robot actions. The approach supports zero-shot control of closed-source frontier VLMs and adaptation of small open-source VLMs with a few GPU-hours of fine-tuning. GUMI (GUI Manipulation Interface) extends the same semantic action space to GUI-based demonstration collection, enabling humans and agents to control robots across embodiments without specialized teleoperation hardware. Experiments show robust generalization across tasks, embodiments, and environments.
Show-Harness introduces a semantic interface that lets VLMs directly control robots by reasoning over discrete action units, with embodiment-specific interpreters translating them into physical actions. It enables zero-shot robot control using closed-source frontier VLMs and low-cost deployment of small open-source VLMs via minimal fine-tuning. The GUMI interface allows GUI-based demonstration collection, removing the need for specialized teleoperation hardware. The system demonstrates robust generalization across tasks, embodiments, and environments.
The key technical contribution is decoupling high-level semantic reasoning from low-level embodiment-specific control. By exposing discrete semantic action units, the VLM remains responsible for fine-grained physical decisions while deterministic interpreters handle grounding. This design allows frontier VLMs to be used zero-shot and small VLMs to be fine-tuned efficiently. GUMI extends the action space to GUI-based demonstration collection, enabling cross-embodiment data gathering without specialized hardware.
This approach lowers the barrier for applying VLMs to robotics by enabling zero-shot control of existing frontier models and cheap fine-tuning of open-source models. The GUI-based demonstration collection could accelerate data acquisition for robot learning across different embodiments, potentially reducing reliance on expensive teleoperation setups.
Show-Harness could reduce development costs for robot control by leveraging existing VLMs and simplifying demonstration collection. It may enable faster prototyping and deployment of robot applications across industries, particularly where specialized teleoperation hardware is costly or unavailable.
Observable next signals include adoption of Show-Harness in robotics research, release of fine-tuned open-source VLM checkpoints, and expansion of GUMI to more embodiments. Further validation on real-world robot platforms and integration with commercial robot systems would indicate progress toward practical deployment.