When Does On-Policy Interaction Help? Representational Tradeoffs in Value-Based Imitation Learning
A research paper introduces OVI, an interactive on-policy imitation learning algorithm that is statistically efficient when the learner can represent the expert's value function. The main finding is that expert interaction relaxes representational demands: the learner only needs to realize the expert's value function, not the full policy. The paper investigates the interplay between interactive querying and value function estimation in imitation learning.
Imitation learning (IL) trains agents to replicate expert behavior from demonstrations. Standard Behavior Cloning suffers from compounding errors and performance plateaus, especially when the learner cannot perfectly represent the expert's policy. Two empirical interventions improve performance: querying the expert interactively along the learner's trajectories, and using value function estimation instead of directly fitting the expert's action distribution. This paper investigates these improvements and their interplay, finding that expert interaction relaxes representational demands: the learner only needs to realize the expert's value function, bypassing the stricter requirement of realizing the expert's policy. The authors introduce OVI, an interactive on-policy IL algorithm that is statistically efficient under this condition.
The key technical insight is that on-policy interaction shifts the representational requirement from policy realizability to value function realizability. This means the learner's model class need only be capable of representing the expert's value function, which is often a simpler function than the full policy. OVI leverages this by using value-based methods in an interactive loop, achieving statistical efficiency when the value function is realizable.
This research could influence how imitation learning is applied in robotics and language model distillation, where perfect policy representation is often infeasible. By reducing representational demands, interactive value-based methods may lower the barrier for deploying IL in complex, real-world systems, potentially improving sample efficiency and performance.
For companies using imitation learning in products (e.g., autonomous systems, AI assistants), this approach could reduce the need for highly expressive policy models, cutting computational costs and improving reliability. It may also enable more efficient use of expert demonstrations, lowering data acquisition expenses.
Next signals to watch include empirical validation of OVI on diverse benchmarks, comparisons with other interactive IL methods, and extensions to settings with partial observability or safety constraints. Adoption in practical distillation pipelines for large language models or robotic control could follow if the theoretical benefits translate to real-world gains.