Policy Iteration with Human Feedback: Bringing Post-Training RL to In-context Learning
A paper titled 'Policy Iteration with Human Feedback: Bringing Post-Training RL to In-context Learning' was published on arXiv on 2026-08-17. It introduces Policy Iteration with Human Feedback (PIHF), which uses a pretrained language model as execution substrate and maintains a versioned natural-language policy and tool set. A language-model critic and clinical expert review trajectories to localize failures and form candidate revisions. Across benchmarks, a PIHF-derived policy improved Recall@1 by 32.7 percentage points for GPT-5.4 and 31.1 points for Qwen3.6-35B, a difference of 1.7 points.
The paper presents PIHF, a method that applies post-training reinforcement learning concepts to in-context learning by iteratively revising a natural-language policy with human feedback. It uses a pretrained language model as the execution substrate, with a critic and expert review to identify failures and propose revisions. The approach was validated on ultra-rare-disease benchmarks, showing significant Recall@1 improvements on multiple executors, including GPT-5.4 and Qwen3.6-35B.
PIHF decouples persistent policy revision from model weights by storing a versioned natural-language policy and tool set, enabling iterative improvement without retraining. The use of a language-model critic for trajectory analysis and expert authority over admission/rollback suggests a hybrid human-AI loop for policy refinement. The reported gains across executors of varying sizes (3B to 49B active parameters) indicate that the policy, not the model scale, drives performance improvements.
This approach could reduce the need for expensive post-training RLHF on large models by shifting adaptation to in-context policy updates, potentially lowering costs and enabling faster iteration. The involvement of clinical experts points to applications in specialized domains like rare disease diagnosis, where high-stakes reasoning benefits from human oversight.
For enterprises, PIHF offers a way to improve model performance on specific tasks without retraining, potentially reducing infrastructure costs and time-to-deployment. In healthcare, it could enhance diagnostic accuracy for rare diseases, leading to better patient outcomes and new product opportunities.
Next signals to watch include whether PIHF is adopted in other domains beyond medicine, whether the versioned policy approach scales to more complex tool-use scenarios, and whether open-weight executors show comparable gains in production settings. Further research may explore automating the critic or reducing expert involvement.