Learning Loco-Manipulation From SMPC Demonstrations With Sparse Offline-to-Online RL
A research paper proposes using Sample-based Model Predictive Control (SMPC) in simulation to generate offline datasets for training reinforcement learning agents with sparse rewards, eliminating manual reward shaping. The method is validated on an arm-equipped Spot quadruped and a G1 humanoid, with learned policies surpassing the original optimal control teacher.
The paper introduces a framework that leverages SMPC as an automated expert to generate massive offline datasets, enabling off-policy RL training with purely sparse task rewards. This approach reduces skill learning time and removes manual tuning. A high-level agent integrated with a low-level dynamic stability controller yields behaviors aligned with true task objectives, and sim-to-real transfer is demonstrated on Spot and G1 robots.
The key technical contribution is using SMPC demonstrations to solve exploration, allowing sparse-reward RL to learn complex loco-manipulation tasks. The offline-to-online RL pipeline with a hierarchical controller (high-level policy + low-level stability) achieves sim-to-real transfer across different robot morphologies.
This method could accelerate deployment of autonomous mobile manipulators by reducing the engineering effort for reward design and enabling rapid skill acquisition. It is relevant to robotics companies developing quadruped and humanoid platforms for industrial or service applications.
The framework lowers the cost and time to train robot manipulation skills, potentially enabling faster product iteration and broader task coverage for commercial robots.
Next observable signals include peer-reviewed publication, code release, or adoption by robotics labs. Further validation on additional tasks and real-world long-horizon deployments would strengthen the approach.