A paper titled 'How to Train a Critic Stably and Efficiently' was published on arXiv on 2026-08-24. It introduces Best-Practice Critic Optimization (BPCO), a recipe combining DPPO, value predictions bounded to the reward range, Monte Carlo value targets, unnormalized policy advantages, and length-adaptive generalized advantage estimation. BPCO improves a critic-based baseline and matches or exceeds a group-based baseline while sampling one response per prompt across mathematical reasoning tasks with models from 1.5B to 30B-A3B MoE.
The paper studies instability in critic-based training for large language models and develops BPCO, a stable and efficient critic optimization recipe. BPCO allows the critic to condition on reward-defining information hidden from the policy, and controlled experiments isolate each design choice. Results show consistent improvement over a strong critic-based baseline and performance matching or exceeding group-based methods with single-response sampling.
BPCO integrates DPPO with bounded value predictions, Monte Carlo targets, unnormalized advantages, and length-adaptive GAE. Conditioning the critic on hidden reward-defining information (e.g., reference answers or rubrics) is a key innovation, enabling token-level advantage estimation from a single response. The approach is validated across model scales up to 30B-A3B MoE.
This work addresses a core efficiency bottleneck in RLHF-style training: reducing the need for multiple sampled responses per prompt. If adopted, BPCO could lower training compute costs and improve stability for LLM alignment, making critic-based methods more competitive with group-based approaches like GRPO.
BPCO offers potential cost savings and stability improvements for LLM training, which could accelerate model iteration and reduce compute expenses for AI labs. It may also enable more efficient use of reward models and rubrics in commercial fine-tuning services.
Next signals to watch include open-source implementations of BPCO, adoption in major LLM training pipelines, and extensions to non-mathematical domains or rubric-based reward settings. Further research may explore scaling to larger models and integration with other RL algorithms.