SUN: Persistent Programs For Language-Grounded Control-to-Learning-to-Real Policies
Kuafu, driven by large vision language systems, automatically synthesizes Semantically UNified (SUN) Programs from language and scene semantics, screens feasibility via MPC, and retains semantics while training stage-conditioned policies. Across nine tasks, Kuafu achieves 82.03% macro-success, outperforming sparse-reward (35.67%) and Stage-BC (24.75%) baselines. At 8192-way scale, it generates 10.57x the successful trajectory time per hour of human teleoperation. With 500 trajectories per task, Kuafu data trains DP3 policies to 46.0% simulation success (vs. 22.4% for alternatives) and 34.7% on physical Franka.
Researchers introduce SUN Programs, typed executables where geometric and contact relations are defined once and compiled into aligned Model Predictive Control (MPC) costs, satisfaction predicates, RL rewards, transition guards, and diagnostics. The Kuafu system uses large vision language models to synthesize SUN Programs from language and scene semantics, screens feasibility via MPC, and retains semantics while training stage-conditioned policies. In experiments across nine tasks, Kuafu achieves 82.03% macro-success, outperforming sparse-reward (35.67%) and Stage-BC (24.75%) baselines. At 8192-way scale, it generates 10.57x the successful trajectory time per hour of human teleoperation. With 500 trajectories per task, Kuafu data trains DP3 policies to 46.0% simulation success (vs. 22.4% for alternatives) and 34.7% on physical Franka.
SUN Programs unify control and learning by compiling a single semantic specification into MPC costs, RL rewards, and transition guards, reducing reward hand-crafting and preserving task semantics during policy training. The use of large vision language systems for automatic program synthesis from language and scene semantics is a key technical enabler. The reported 10.57x improvement in successful trajectory time per hour of human teleoperation at 8192-way scale suggests significant data collection efficiency gains.
This work addresses a practical bottleneck in robot learning: the need for large amounts of high-quality demonstration data. By generating semantically aligned trajectories at scale, it could reduce the cost and time required to train manipulation policies, making robotic automation more accessible for industrial and research applications.
The technology could lower the barrier to deploying robotic manipulation systems by automating the generation of control and learning specifications from natural language, reducing the need for expert reward engineering and manual data collection. This may accelerate adoption in manufacturing, logistics, and service robotics.
Next observable signals include whether the approach scales to more diverse tasks and environments, whether the synthesized SUN Programs generalize across different robot platforms, and whether the trained policies can be deployed in real-world settings beyond the reported Franka experiments. Further validation on physical robots and integration with commercial robot learning pipelines would be key indicators of progress.