What Matters, When? Diagnosing and Improving Conditional Visual Grounding in Visuomotor Imitation Policies
A study on visuomotor imitation policies finds that high in-distribution performance can fail when visually similar objects or receptacles are introduced. Using Action Chunking with Transformers (ACT), the authors systematically introduce distractor objects and receptacles with controlled color and shape similarity, localizing failures to picking and placement. They evaluate distractor augmentation, phase-dependent attention regularization, and appearance-based visual prompting, reporting substantial robustness improvements in simulation and on a physical UR3e. The same failure pattern is examined in a pretrained vision-language-action policy on a state-conditioned instrument-handling task.
Researchers diagnose conditional visual grounding failures in visuomotor imitation policies, showing that target selection depends on manipulation phase and task state. They introduce controlled distractors and find sensitivity specific to visual similarity type and manipulation stage. Three interventions—distractor augmentation, phase-dependent attention regularization, and appearance-based visual prompting—improve robustness in simulation and on a UR3e robot. The pattern is also observed in a pretrained vision-language-action policy.
The work isolates conditional visual grounding as a key failure mode: the visual target required for control changes with manipulation phase and task state. Interventions such as phase-dependent attention regularization and appearance-based visual prompting aim to preserve spatial information while improving target selection. Evaluation on a physical UR3e suggests transferability beyond simulation.
Robustness to visual distractors is critical for deploying visuomotor policies in unstructured environments. The proposed interventions could reduce failure rates in industrial manipulation tasks where objects and receptacles vary in appearance. The observation of similar failures in a pretrained vision-language-action policy indicates broader relevance for general-purpose robot policies.
Improved robustness to visual distractors can lower error rates and downtime in robotic manipulation, potentially reducing integration costs for industrial automation. The techniques may be applicable to existing visuomotor policies, offering a path to more reliable deployment without full retraining.
Next signals include whether these interventions scale to more complex, multi-stage tasks and whether they can be integrated into pretrained vision-language-action models without retraining. Further validation on diverse robot platforms and real-world settings would strengthen the case for adoption.