One-Step Text Generation with Seq-Drifting: Diffusion LMs Break Multi-Step Barrier

From Multi-Step Denoising to One-Step Generation

Current autoregressive language models generate tokens sequentially, creating serial dependency that slows long outputs. Diffusion language models parallelize token updates but still require multiple denoising rounds to coordinate token relationships. Reducing steps to one challenges consistency: a token plausible in isolation may conflict with others. Seq-Drifting directly trains a mapping from Gaussian noise to a full token sequence in one forward pass.

Context-Dependent Drifting Field

The core innovation is a context-dependent drifting field. During training, the generator produces continuous token representations in parallel. Based on the currently generated prefix, a candidate set of compatible tokens is constructed for each position, providing attraction targets that adapt as the prefix changes. For example, given prefix "She put the book on", candidates for the next position include "the table" or "the shelf"; if the prefix becomes "She gave the book to", candidates shift to objects that can receive a book.

Two candidate construction methods are used: (1) n-gram retrieval from the training corpus to find real texts matching the current generated fragment, and (2) a frozen support model (e.g., GPT-2) that scores next-token probabilities conditioned on the generator's current prefix. Attraction pulls representations toward compatible candidates; repulsion between samples encourages diversity, and repulsion within a sequence suppresses token repetition. These forces are combined on the unit hypersphere, matching cosine-similarity-based token recovery.

Training and Inference Pipeline

Training starts from Gaussian noise. The generator outputs all token representations in one forward pass, normalizes them to the unit sphere, computes candidate sets via retrieval or support model, calculates attraction/repulsion forces, performs a bounded spherical update to obtain per-position targets, and regresses the generator toward these targets. Ablation shows removing intra-sequence repulsion causes repetition; removing spherical constraint leads to divergence.

At inference, only the trained generator is used: noise (and optional condition) are fed in, one forward pass yields all representations, which are mapped to discrete tokens. The support model is not needed at inference. For conditional generation, the generator receives a given prefix via cross-attention; the support model constructs candidates using both the prefix and the generated continuation. Training begins with unconditional warm-up, then introduces real prefixes to learn conditional dependencies.

Quality and Efficiency Results

Evaluated on LM1B (128 tokens) and OpenWebText (1024 tokens). Two settings: S-version uses a support model trained from scratch on the same corpus; P-version uses public GPT-2. Both generators run one forward pass at inference.

LM1B: S-version Gen-PPL 93.81 (vs. 4-step FMLM 98.76); P-version Gen-PPL 36.01.

Speed: 11.2x faster than 4-step FMLM on LM1B, 10.1x on OpenWebText.

Token entropy and Self-BLEU confirm diversity.

Conditional continuation: both versions outperform few-step distillation baselines.

Translation, Summarization, and Reasoning

Tested on WMT14 German-English translation and XSum summarization. One-step generation beats few-step baselines on most metrics but lags behind stronger multi-step models. Scaling up the support model improves one-step quality, suggesting better supervision from larger LMs.

Reasoning tasks (SVAMP, ASDiv math; ProofWriter logic) with 0.5B-parameter generators show competitive accuracy and low per-sample latency (milliseconds). The team plans to scale model and training size for further reasoning exploration.

Future Directions

Seq-Drifting demonstrates a viable path to direct one-step text generation: training aligns token choices via context-dependent targets; inference produces the full sequence in one forward pass. Future work will explore stronger support models, richer semantic supervision, more reliable token recovery, and larger-scale mathematical reasoning evaluation. Code and weights are publicly available: https://github.com/jyliu-ai/Seq-Drifting, https://huggingface.co/jyliuAI/Seq-Drifting, paper: https://arxiv.org/abs/2610.12879.