Skip to content
Finer Sampling Steps Boost Privacy Leakage in Diffusion LLMs, NeurIPS 2026 Finds

Finer Sampling Steps Boost Privacy Leakage in Diffusion LLMs, NeurIPS 2026 Finds

Aalborg University researchers present a generalized extraction framework showing diffusion language models leak less training data than autoregressive models at same scale, but finer sampling steps significantly increase exact memorization recovery, with LLaDA-8B showing a 20x increase in email extraction when switching from one-step to token-by-token generation.

Machine Heart
Machine Heart
Machine Heart

Introduction

Large language models sometimes reproduce training content verbatim, potentially exposing personal information such as emails and phone numbers. As diffusion language models scale up, a key question arises: when models no longer generate left-to-right token by token, do they leak training data in the same way?

Autoregressive models predict the next token given previous context, while masked diffusion language models operate more like fill-in-the-blank, iteratively completing missing content from masked text. This parallel generation capability changes the pathway through which training data can be extracted.

Generalized Probabilistic Extraction Framework

Researchers from Aalborg University, in their NeurIPS 2026 paper "Characterizing Memorization in Diffusion Language Models: Generalized Extraction and Sampling Effects" (arXiv:2603.02333), propose a generalized probabilistic extraction framework. Given partially visible text, the framework allows the model multiple attempts and measures the probability of recovering all masked original content. This framework supports arbitrary masking positions, different recovery orders, and subsumes traditional prefix continuation as a special case.

The feasibility of probability estimation was validated: on 200 samples, estimates from 1,024 random mask samplings closely matched empirical results from 100,000 random generations, providing a basis for reducing evaluation overhead.

Figure 2: Probability estimates vs. empirical extraction probabilities from large-scale generation (original paper Figure 2).
Figure 2: Probability estimates vs. empirical extraction probabilities from large-scale generation (original paper Figure 2).

Why Finer Sampling Steps Increase Leakage

The process can be likened to filling in a familiar passage: completing all blanks at once gives each position limited clues; but if a few positions are filled correctly first, those new clues make the remaining content easier to recover.

The paper provides a theoretical characterization: under the assumption that adding correct context does not decrease the recovery probability of the remaining content, fixing the recovery order and splitting the original batch into finer steps does not decrease the probability of full recovery.

Crucially, this conclusion concerns the relationship between correct context and specific step refinement, and cannot be directly interpreted as "more steps always mean worse leakage" under any sampling strategy.

Experiments confirm the trend. The team selected 200 mask configurations meeting filtering criteria and tested one-step, two-step, five-step, ten-step, and token-by-token recovery settings with extensive generation. As steps became finer, the success rate of exact recovery increased overall.

The model parameters did not change; what changed was the opportunity for existing memories to be extracted. Sampling steps therefore affect not only generation speed and quality but also the extractability of training data.

Figure 3: Finer sampling steps increase exact recovery success rate relative to one-step generation; shaded areas show 95% confidence intervals (original paper Figure 3).
Figure 3: Finer sampling steps increase exact recovery success rate relative to one-step generation; shaded areas show 95% confidence intervals (original paper Figure 3).

Diffusion vs. Autoregressive Models on Email Completion

To compare the two model classes, the team trained a 1.1B-parameter diffusion model and a 1.1B-parameter autoregressive model with aligned datasets, tokenizers, and compute budgets. Both were adapted using the same Enron email data pipeline.

Evaluation gave models the preceding 100 tokens of a target email or phone number and asked them to complete the target. Prompts containing other email addresses were filtered to reduce extra clues.

Under a budget of 3,000 email completion samples and up to 10,000 queries per sample, two results stand out:

At 1.1B parameters, even with token-by-token recovery, the diffusion model reached a 50% extraction probability threshold on only 16 email samples, while the autoregressive model reached it on 213 samples.

For LLaDA-8B, switching from one-step to token-by-token recovery increased the number of samples reaching the 50% threshold from 9 to 179 — nearly a 20-fold increase. The same model, different sampling settings, vastly different risk.

However, these results correspond to aligned prefix completion tasks and do not prove diffusion models are safer in all scenarios, especially their native bidirectional fill-in-the-blank setting.

Table 1: Samples reaching extraction probability thresholds under given query budget (reproduced from original paper Table 1).
Table 1: Samples reaching extraction probability thresholds under given query budget (reproduced from original paper Table 1).

Memorization vs. Guessing

Successfully completing an email does not necessarily prove memorization; common formats and language patterns could allow the model to guess parts correctly. To disentangle this, the team compared Enron emails seen during training with unseen TREC 2007 Spam emails from the same domain, drawing 10,000 text windows each and evaluating reconstruction probability.

The reconstruction probability distribution for training data was overall higher, indicating the evaluation captures differences related to training exposure and supporting the claim that extraction results contain a memorization effect.

Lower extraction rates do not mean the model has not memorized; they may mean those memories are harder to extract under current prompting and sampling conditions.

Figure 4: Reconstruction probability distributions for training (Enron, blue) vs. unseen (TREC, orange) emails (original paper Figure 4).
Figure 4: Reconstruction probability distributions for training (Enron, blue) vs. unseen (TREC, orange) emails (original paper Figure 4).

Three Implications for Privacy Evaluation

Evaluate the generation setup, not just the model. The same parameters can exhibit different training data extraction risks under different sampling step counts.

A single miss does not mean multiple queries cannot extract. Query budget, success probability, and masking strategy should all be reported.

The lower leakage of diffusion models warrants further study, but boundaries need verification. Whether the property holds after bidirectional fill-in-the-blank, instruction tuning, and preference optimization remains to be tested.

This work introduces an often-overlooked variable into model safety assessment: training determines what the model has seen, but the generation process influences how easily that content can reappear.

Original Source

Signed-in readers can open the original source through BestHub's protected redirect.

Sign in to view source
Republication Notice

This article has been distilled and summarized from source material, then republished for learning and reference. If you believe it infringes your rights, please contactadmin@besthub.devand we will review it promptly.

autoregressive modelsdiffusion language modelsmemorizationLLaDANeurIPS 2026privacy leakagesampling steps
Reader feedback

How this landed with the community

Sign in to like

Rate this article

Was this worth your time?

Sign in to rate
Discussion

0 Comments

Thoughtful readers leave field notes, pushback, and hard-won operational detail here.