A self-supervised method called Structure-Aware Fine-Tuning (SAFT) refines noisy VLM-based reward signals online using LoRA adapters and intrinsic structural priors, without ground-truth supervision. Evaluated across various base model capabilities, SAFT denoises the reward landscape, leading to faster policy convergence and improved alignment measured by EPIC distance.
Designing reward functions for reinforcement learning is a bottleneck. Recent work uses large vision-language models (VLMs) as reward models by computing text-observation similarity, but these rewards are often noisy and unreliable. SAFT is a self-supervised method that refines these imperfect reward signals online without ground-truth supervision. It leverages intrinsic structural priors to regularize the VLM's latent space via LoRA adapters. Evaluated across a spectrum of base model capabilities, SAFT consistently denoises the reward landscape, yielding faster policy convergence and substantially improved alignment (EPIC distance) relative to the underlying base model. The results suggest that failures can often be attributed to structural brittleness rather than semantic misunderstanding. By replacing extensive human preference annotation with structural inductive biases inherent to the task, SAFT offers a scalable path for improving VLM reward models.
SAFT applies LoRA adapters to regularize the VLM's latent space using intrinsic structural priors, effectively denoising reward signals without ground-truth labels. This indicates that structural brittleness, not semantic misunderstanding, is a key failure mode in VLM reward models.
The approach reduces reliance on costly human preference annotation, potentially lowering the barrier for deploying RL with VLM rewards in robotics and embodied AI. Next signals to watch include adoption of SAFT in real-world robotic manipulation tasks and integration with other VLM architectures.
By automating reward refinement, SAFT reduces the need for manual reward engineering and human annotation, cutting development costs and time for RL-based applications in robotics, gaming, and simulation.
SAFT could enable more robust and scalable reward modeling for RL, accelerating progress in autonomous systems. Future work may explore combining SAFT with other self-supervised techniques or extending it to multi-modal reward models beyond VLMs.