MissClick: Exploiting Digit-Serialized Coordinates to Attack GUI Grounding Models
A white-box adversarial attack named MissClick targets GUI visual grounding models that generate screen coordinates as sequences of digit tokens. The attack exploits the numerical structure of coordinate outputs, where changing a hundreds-place digit by one can displace the executed click by 100 units. Two objectives are proposed: MissClick-U maximizes soft-coordinate displacement for untargeted disruption, and MissClick-T minimizes a place-weighted target-digit loss for targeted hijacking.
Researchers have identified a vulnerability in GUI visual grounding models that generate screen coordinates as digit sequences. The MissClick attack manipulates these digit tokens, leveraging the fact that a single digit change in the hundreds place can shift a click by 100 pixels. The attack offers two variants: untargeted disruption (MissClick-U) and targeted hijacking (MissClick-T). The paper was published on arXiv on August 4, 2026.
The attack treats coordinate generation as a categorical token prediction problem but exploits the numerical semantics of the output. By designing loss functions that account for place value and coordinate displacement, MissClick achieves effective adversarial perturbations. MissClick-U uses a soft-coordinate displacement maximization objective, while MissClick-T uses a place-weighted target-digit loss to steer clicks to attacker-chosen regions.
As GUI automation agents become more prevalent, adversarial vulnerabilities in their grounding components could be exploited to hijack user interactions or disrupt automated workflows. This research highlights a new attack surface in multimodal models that output structured numerical data, which may require novel defenses beyond standard text-based adversarial robustness.
The research underscores security risks for enterprises deploying GUI automation agents, potentially impacting trust and adoption. It may drive demand for robustness testing tools and secure model design practices, creating opportunities for security-focused AI startups and influencing procurement requirements.
Future work may focus on developing defenses against coordinate-based attacks, such as input sanitization, robust parsing, or adversarial training. The attack's effectiveness may also be tested on commercial GUI agents, and similar vulnerabilities could be explored in other domains where models output structured numerical sequences.