A study audits three rounds of rank-32 LoRA self-training on Qwen3-8B against a frozen control. It identifies seven measurement failures, each of which inverts a reported finding when its control is absent. A ledger built on a single greedy decode manufactures capability changes on an untrained model, largely an artifact of inference batching. The expansion statistic separating acquisition from sharpening assigns that same model a rate of 0.280. The natural threshold repair does not survive replication; its null stays non-zero. A per-problem exact test against a pooled baseline under false-discovery-rate control detects nothing on any held-out replicate and is unchanged under the multiple-testing rule, error rate and pool size.
The paper 'Phantom Gains: Auditing Self-Improvement Against a Measured Null' examines whether language model self-improvement is real or an artifact of measurement. By comparing self-trained Qwen3-8B models against a frozen control, the authors find that common evaluation practices can create false signals of improvement. They propose a more rigorous statistical test that eliminates these phantom gains.
The study highlights the importance of control conditions in evaluating self-improvement. It shows that inference batching can introduce artifacts in greedy decoding, and that naive threshold-based metrics fail to replicate. The proposed per-problem exact test with false-discovery-rate control provides a more reliable method for detecting genuine capability changes.
This research suggests that many reported gains from self-training or self-improvement in language models may be overstated. Practitioners should adopt rigorous controls and statistical tests to avoid chasing phantom improvements, which could lead to wasted resources and incorrect model selection.
For AI companies, this paper underscores the risk of investing in self-improvement techniques based on flawed metrics. Adopting the proposed auditing methods could save costs and improve model quality by focusing on real gains.
Future work may focus on developing standardized evaluation protocols for self-improvement, including mandatory control conditions and robust statistical methods. This could lead to more reliable benchmarks and a clearer understanding of when self-training genuinely helps.