Selective Agent Guidance via Entropy: Learning Autonomous Policies from Imperfect VLM Teachers
SAGE (Selective Agent Guidance via Entropy) is a framework that queries a Vision-Language Model (VLM) only when the learner is uncertain, executes the suggested action during training, and distills guidance into a lightweight Reinforcement Learning (RL) policy. Across sparse-reward visual reasoning and navigation tasks, SAGE learns policies that act without VLM guidance at evaluation time and improves over unguided RL in several environments, including settings where the learned policy exceeds its VLM teacher.
Researchers propose SAGE, a framework for learning autonomous policies from imperfect VLM teachers. SAGE selectively queries a VLM based on learner uncertainty, uses the suggested action during training, and distills guidance into a lightweight RL policy. The approach improves over unguided RL in several sparse-reward visual reasoning and navigation tasks, and can even exceed the VLM teacher's performance.
SAGE uses entropy-based uncertainty to decide when to query the VLM teacher, reducing expensive and brittle per-step queries. It weights teacher-action distillation using environment-derived advantages, mitigating the impact of unreliable advice. The learned policy operates without VLM guidance at evaluation time, demonstrating effective knowledge transfer from an imperfect online teacher to a lightweight autonomous agent.
This research addresses the practical challenge of deploying VLMs in interactive decision-making by reducing inference cost and improving robustness. The ability to distill VLM knowledge into lightweight policies could enable more efficient and reliable autonomous systems in robotics and visual reasoning applications.
SAGE offers a cost-effective way to leverage VLM capabilities for autonomous decision-making without continuous VLM inference. This could reduce operational costs and improve performance in robotics, navigation, and other interactive AI systems, making VLM-guided policies more commercially viable.
Future work may explore scaling SAGE to more complex environments, improving uncertainty estimation, and applying the framework to other teacher models. The approach could lead to more sample-efficient RL and broader adoption of VLM-guided policy learning in real-world applications.