ReflectRL: Learning from Golden Negative Trajectories via Reflective-to-Direct Reasoning
On-policy training is a post-training paradigm for improving LLM reasoning, often using golden trajectories from stronger expert models. When the expert fails on harder problems, these failed trajectories are typically discarded. The paper argues that such Golden Negative Trajectories can provide valuable reasoning signals when used for reflection rather than imitation. It identifies a Reflection Advantage: reflecting on a flawed trajectory can be easier and more effective than solving directly. ReflectRL is a lightweight plug-and-play framework that learns from Golden Negative Trajectories during on-policy training, first eliciting Reflective Reasoning and then applying Reflective-to-Direct Policy Transition to transfer reasoning behavior back to Direct Reasoning. Experiments cover 9 benchmarks, 4 LLM backbones, and 4 on-policy methods.
A new framework, ReflectRL, leverages previously discarded expert failure trajectories (Golden Negative Trajectories) to improve LLM reasoning. Instead of imitating these flawed paths, the model reflects on them to learn better reasoning strategies, then transfers this skill to direct problem-solving. This approach addresses the limitation of on-policy training when expert demonstrations are unavailable for hard problems.
The core innovation is the Reflection Advantage hypothesis: for difficult problems, analyzing a flawed solution is more tractable than generating a correct solution from scratch. ReflectRL operationalizes this via a two-stage process—first training the model to critique and correct negative trajectories (Reflective Reasoning), then distilling this capability into the standard direct reasoning policy through a policy transition mechanism. This decouples learning from expert success and expands the usable training signal.
This research could reduce reliance on expensive human or model-generated golden trajectories for post-training, potentially lowering the cost and increasing the scalability of improving LLM reasoning. It may enable more efficient use of existing model outputs, including failures, which are abundant in real-world deployments.
By turning failed attempts into training data, ReflectRL can improve model performance on hard problems without additional expert annotation costs. This could accelerate the development of more capable reasoning models for applications in math, coding, science, and strategic planning, offering competitive advantage to adopters.
If validated across diverse tasks and models, ReflectRL could become a standard component in LLM training pipelines. Next signals to watch include open-source implementations, integration into popular RLHF/on-policy frameworks, and extensions to multi-modal or agentic reasoning scenarios.