Tencent open-sourced UI-Mate-democua-27B, a demonstration-guided GUI agent

UI-Mate-democua-27B is a demonstration-guided checkpoint of UI-Mate, an open-weight foundation GUI agent. It observes live screenshots, reasons over the visible state, and produces structured keyboard and mouse actions. It can take one recorded workflow as input and carry that procedure over to a new task. The model is trained with supervised fine-tuning on a mixture of general computer-use data and demonstration-augmented data, preserving instruction-only competence while adding demonstration-guided execution. A demonstration is treated as guidance rather than a fixed action script; recorded coordinates are never replayed, and the model re-plans from the live interface when content, layout, or application state differs. The model has 27B parameters, uses Qwen3.6-27B as its base, and is licensed under Apache-2.0. It is an agent checkpoint rather than a standalone visual-chat model, and Tencent recommends using the official prompt, response parser, and interaction harness from the UI-Mate repository.

The model uses a demonstration-guided execution approach where a recorded workflow is normalized, annotated by a vision-language model along four axes (screen state, intent, action taken, target location), segmented into named subtasks with checkable completion criteria, and supplied at inference as a compact view of the active subtask. Training deliberately withholds part of the guidance (key actions like focus clicks and scrolling are omitted) so the model must infer missing steps from screenshots. The training mixture covers full alignment, partial misalignment, and irrelevance between guidance and screen, with full alignment as the majority case. At inference, the complete action sequence of the current subtask is passed without key-action extraction. Evaluation on OSWorkerBench-Subset (33 tasks) shows strict success improving from 17.17 to 35.35 (+18.18 pp) and progress from 67.85 to 81.14 (+13.29 pp) when adding one demonstration. On OSWorld-Subset (30 tasks), progress improves from 40.27 to 65.75 (+25.48 pp). On GameDev (10 tasks), average score improves from 76.76 to 81.15 (+4.39 pp).