From Truncation to Commitment: Persistent Context in Uniform Discrete Diffusion
The paper proposes committed reveal sampling (CRS), a training-free sampler for uniform-state discrete diffusion models that stores selected argmax tokens and inserts them into subsequent model inputs. Under the exact forward process, the Bayes error of selecting a clean token cannot increase as noise decreases. In a simple latent-mode model, keeping the selected token visible helps later parallel predictions agree on the same sequence-level choice. Paired experiments on Duo-distilled models show that CRS without top-p truncation reaches lower generative perplexity (GenPPL) than fixed p=0.95 and p=0.9 baselines.
Uniform-state discrete diffusion models update all tokens in parallel while keeping every position revisable. Even when the commonly used top-p rule leaves only one candidate at a position, that choice affects only the current reverse step and can be revised at the next sampling step. The paper asks what changes when selected hypotheses instead become persistent context for later predictions. It proposes committed reveal sampling (CRS), a training-free sampler that stores selected argmax tokens and inserts them into subsequent model inputs. The analysis gives a rationale for selecting later and for keeping selected tokens visible. Under the exact forward process, the Bayes error of selecting a clean token cannot increase as noise decreases, while in a simple latent-mode model, keeping the selected token visible helps later parallel predictions agree on the same sequence-level choice. Empirically, paired experiments on Duo-distilled then separate this persistent effect from single-step top-p restriction and scalar temperature scaling. Under the same finalization rule, CRS without top-p truncation reaches lower generative perplexity (GenPPL) than fixed p=0.95 and p=0.9 baselines.
CRS introduces a persistent context mechanism in discrete diffusion sampling by committing to argmax tokens and feeding them back into the model, which reduces revisability and improves sequence-level consistency. The theoretical result that Bayes error is non-increasing with noise reduction supports later commitment, while the latent-mode model shows that persistent visibility aligns parallel predictions. The empirical GenPPL improvement over top-p baselines suggests that persistent context can outperform truncation-based finalization without additional training.
This work addresses a practical limitation in discrete diffusion models: the inefficiency and inconsistency of parallel token updates. By making selected tokens persistent, CRS could improve the quality and coherence of generated text or code, potentially making discrete diffusion more competitive with autoregressive models. The training-free nature lowers adoption barriers for existing discrete diffusion pipelines.
For organizations using discrete diffusion models for text, code, or biological sequence generation, CRS offers a simple, training-free method to improve output quality (lower GenPPL) without additional compute or data. This could reduce the need for expensive fine-tuning or larger models, and may accelerate deployment of discrete diffusion in production systems.
Next signals to watch include whether CRS is adopted in open-source discrete diffusion implementations, whether it generalizes to other discrete diffusion variants (e.g., absorbing-state or multinomial diffusion), and whether it enables faster sampling with fewer steps while maintaining quality. Further theoretical work may extend the latent-mode analysis to more complex sequence dependencies.