SpanUQ: Span-Level LLM Uncertainty Estimation 10-20x Faster Than Sampling

01 | From Token to Sequence: Granularity Limits of Existing Uncertainty Estimation

In real-world LLM product development at Amazon, downstream systems need not only generated results but also confidence information to optimize decisions—whether to accept output, request verification, or call a stronger model. This requires Uncertainty Estimation . A key question arises: when a generated text contains multiple facts or semantic units, can a single score measure the reliability of the entire response?

If a model produces ten factual claims with nine reliable and one uncertain, an average confidence score is insufficient for fine-grained decisions. We need to know exactly which content the model is confident about and which it is not. This becomes even more critical in future Multi-Agent systems where agents exchange information; passing uncertainty signals per claim would allow receiving agents to decide on verification, further reasoning, or strategy adjustment. Thus, uncertainty should be a decision signal usable by downstream systems, not just a model evaluation metric. This leads to a fundamental question: at what granularity should uncertainty be measured?

1.1 | Token-Level Uncertainty: Fine-Grained but Lacks Complete Semantics

Token-level methods use per-token probability distributions (entropy, token probability) to estimate uncertainty. They have low computational overhead and provide fine-grained information. However, a token does not naturally correspond to a semantically complete unit . For example, a date, number, or entity name may consist of multiple tokens; individual token uncertainty does not directly reflect the reliability of the whole factual claim. Moreover, probability uncertainty does not equal factual uncertainty: a model may confidently generate a false fact, or show high token entropy for correct content due to multiple valid phrasings. Using token-level uncertainty directly for downstream factual reliability judgment can introduce bias.

1.2 | Sequence-Level Uncertainty: Holistic View but Cannot Locate Issues

Sequence-level methods estimate uncertainty for the entire response, e.g., via multiple sampling, semantic consistency comparison, or other techniques to compute a single uncertainty score. While they measure overall reliability, they cannot pinpoint which part is problematic. Consider the statement: "Marie Curie was a Polish physicist who won the Nobel Prize in 1901." It contains three facts: (1) Marie Curie is a Polish physicist (correct), (2) She won a Nobel Prize (correct), (3) She won in 1901 (incorrect; actual year 1903). A sequence-level method yields only one score for the whole response, but downstream systems care about the specific erroneous fact, not rejecting the entire answer. This motivates the search for a granularity between token and sequence.

02 | Span-Level Uncertainty Estimation: Aligning Uncertainty with Semantics

Core observation: humans understand, verify, and use information in units of semantically complete content fragments, not tokens or whole texts. For a text with multiple factual claims, we can judge each fact's reliability independently. Based on this, the paper formally proposes a new research task:

2.1 | Span-Level Uncertainty Estimation (SLUE)

SLUE aims to identify text spans with complete, independently evaluable semantics (Semantic Spans) in LLM-generated text and estimate uncertainty for each span . Compared to existing granularities, span-level estimation targets three properties:

Semantic Coherence : each span corresponds to a semantic unit that can be independently understood and evaluated, not an arbitrary token combination.

Fine-Grained Localization : when only part of a response is uncertain, precisely locate those spans rather than scoring the whole response.

Actionable Uncertainty : linking uncertainty to specific semantic content enables downstream systems to apply different strategies per span—e.g., verify only high-uncertainty claims without re-checking the entire response.

The third point is especially important: it turns uncertainty estimation from a model evaluation task into a bridge connecting LLM generation with downstream decisions. SLUE does not replace token-level or sequence-level uncertainty; different granularities reflect different information layers, and SLUE complements existing methods in semantic localization and actionability.

2.2 | From Problem Definition to a Researchable Task

Proposing a new task is only the first step. Real challenges: how to automatically identify semantic spans? How to construct reasonable uncertainty labels for different spans? How to estimate without significantly increasing inference cost? The paper introduces SpanUQ as an initial solution for SLUE.

03 | SpanUQ: Single Forward Pass for Span-Level Uncertainty Estimation

SpanUQ addresses two key problems: (1) how to locate semantic spans needing evaluation, and (2) how to efficiently and accurately estimate their uncertainty. Existing sampling-based methods require multiple generations per query, incurring high inference overhead. SpanUQ asks: can we distill expensive multi-sample uncertainty information into a lightweight model that predicts via a single forward pass?

SpanUQ is a ~25M parameter lightweight probe that uses frozen pre-trained LLM hidden states to jointly perform semantic span detection and uncertainty estimation. The framework comprises:

3.1 | Extracting Uncertainty Information from Hidden States

LLM internal representations contain rich information; different layers may encode different semantic and predictive features. Hidden states are extracted from the pre-trained LLM and transformed via Multi-Layer Fusion and Token Encoder into features suitable for span-level prediction. The base LLM parameters remain frozen—no full retraining needed. The probe learns a mapping from hidden states to target uncertainty labels, not reading an existing explicit factual-correctness variable.

3.2 | DETR-Style Span Decoder: Parallel Detection of Multiple Semantic Spans

A naive approach would frame span detection as token classification (e.g., BIO tagging), but this struggles with overlapping spans, flexible boundaries, and joint prediction. Inspired by the DETR architecture in object detection, the authors design a DETR-Style Span Decoder . It uses a set of learnable Span Queries that interact with text features via cross-attention to predict multiple semantic spans in parallel. Each query predicts a candidate span's position and its uncertainty. This design frees span detection from token-by-token labeling and provides a flexible structure for overlapping semantic spans. In experiments, the Span Detector achieves 0.910 F1 , a 39.4% relative improvement over the strongest heuristic baseline, demonstrating the viability of query-based prediction for semantic span localization.

3.3 | Mixture of Beta: Modeling Continuous Uncertainty

After locating a span, how to predict its uncertainty? Simple regression of a 0–1 score is insufficient because uncertainty distributions across spans can be complex (some highly reliable, some clearly unreliable, some intermediate). The authors employ a Mixture of Beta Distributions (MoB) . Beta distributions are naturally defined on [0,1], suitable for bounded continuous values. Combining multiple beta components expresses complex uncertainty distributions beyond a single point estimate. Training combines Beta Negative Log-Likelihood (Beta NLL) for learning continuous uncertainty labels and Contrastive Ranking Objectives to encourage correct relative uncertainty ordering across spans. Both capabilities matter: downstream systems need not only accurate scores but also reliable ranking of content reliability.

3.4 | Uncertainty-Conditioned Iterative Refinement (UCIR)

An Uncertainty-Conditioned Iterative Refinement (UCIR) module feeds first-round uncertainty predictions back into the shared decoder to refine span predictions. This adds limited extra computation ( overhead < 15% ) and further improves uncertainty ranking correlation.

Overall, SpanUQ unifies Span Detection and Uncertainty Quantification in a single framework, showing that fine-grained, semantically grounded uncertainty signals can be obtained without expensive multi-sample inference.

04 | SpanUQ-Bench: Standardized Evaluation Foundation for SLUE

A new research task requires unified data and evaluation. The authors construct SpanUQ-Bench as a preliminary benchmark for SLUE, containing:

20K Prompts

~293K Annotated Spans

Continuous Uncertainty Labels

Unlike traditional binary correctness labels, continuous soft labels capture each span's uncertainty degree across multiple generations and verifications. Using Multi-Sample Claim Verification , if a factual claim shows high consistency across multiple samples and verifications, it receives low uncertainty; high inconsistency or lack of stable verification support yields high uncertainty. While multi-sample empirical signals are not absolute factual correctness probabilities, they provide valuable supervision for learning fine-grained uncertainty estimation. SpanUQ-Bench aims to turn SLUE from a conceptual question into a trainable, evaluable, and comparable task, and hopes to inspire broader benchmarks covering more task types, languages, model architectures, and uncertainty definitions.

05 | Experimental Results: Accuracy, Efficiency, and Model Applicability

Experiments on five LLM backbones across two model families (Dense and Mixture-of-Experts) evaluate span-level uncertainty quality, span detection, and inference efficiency.

5.1 | More Accurate Span-Level Uncertainty Estimation

On all five backbones, SpanUQ achieves strong performance: AUROC 0.908–0.944 , MAE 0.110–0.129 . Compared to baselines (Token Entropy, MLP Probe, SelfCheckGPT-NLI, P(True)), SpanUQ shows superior comprehensive performance. The paper emphasizes both discrimination of high/low uncertainty spans and continuous prediction error (MAE), which is crucial for downstream systems using uncertainty scores for decisions.

5.2 | 10–20× Inference Speedup

Efficiency is a key advantage. Traditional sampling-based methods require multiple model calls to generate candidate answers. SpanUQ uses LLM hidden states with a lightweight probe for prediction. Under the paper's evaluation setup, SpanUQ achieves ~10–20× inference speedup over sampling-based methods, demonstrating that fine-grained uncertainty estimation need not incur high multi-sampling costs.

5.3 | From Span-Level to Sequence-Level

The authors explore whether span-level uncertainties can be composed to derive sequence-level uncertainty. Results indicate that sequence-level uncertainty is partially decomposable, suggesting span-level estimation not only provides finer information but may also help understand higher-level uncertainty. This hints at structural relationships between uncertainty granularities worth further modeling.

06 | From Uncertainty Estimation to Uncertainty-Aware Decision Making

Beyond performance gains, the long-term interest is: what can we do when LLMs provide uncertainty signals for specific semantic spans? Three promising directions:

6.1 | Uncertainty-Aware Verification: Targeted Verification

Fact verification of entire responses is costly. If only two of ten claims have high uncertainty, downstream systems can invoke search tools or verification models only for those spans, concentrating compute where needed. Different uncertainty levels could map to different verification strategies: low uncertainty → direct use; medium → lightweight verification; high → strict external retrieval or cross-model verification. Low uncertainty does not guarantee factual correctness, so strategies must consider task risk, model calibration, and business needs. Nevertheless, span-level uncertainty provides a potential foundational signal for more fine-grained verification strategies .

6.2 | Uncertainty-Guided Self-Refinement: Modify Only What Needs Modification

Self-refinement typically regenerates the whole response. If only a few spans are problematic, full regeneration wastes compute and may alter correct content. Span-level uncertainty enables locating high-uncertainty spans and guiding the model to verify, modify, or regenerate only those spans—moving from whole-response refinement to targeted span-level refinement. Open questions: how to decide if a span needs refinement based on uncertainty? How to maintain context consistency when editing locally? How to verify that refinement actually reduces error risk? These require further research, but targeted span-level refinement is a promising direction .

6.3 | Uncertainty-Aware Multi-Agent Communication: Exchanging Reliability Signals

In multi-agent systems, agents collaborate on complex tasks. Currently agents exchange natural language but rarely communicate reliability of that information. If Agent A provides uncertainty scores per claim alongside its analysis, Agent B can decide whether to use the information directly, request evidence, or call another agent for cross-verification. The whole system could dynamically allocate tasks, compute, and reasoning paths based on per-claim uncertainty. This suggests a broader interaction paradigm: future agent communication may include reliability signals attached to specific semantic units, not just natural language text . Open challenges: comparability of uncertainty scores across different base models, ensuring calibration consistency, preventing propagation of miscalibrated signals. Span-level uncertainty is not a complete solution but a starting point for investigating these problems.

07 | Conclusion: A Just-Beginning Research Direction

As LLMs evolve from standalone text generators into core components of complex AI systems, measuring, understanding, and communicating output reliability becomes increasingly important. This work starts from a basic question: what is the appropriate granularity for uncertainty estimation? Token-level uncertainty lacks complete semantics; sequence-level uncertainty cannot locate problematic content. Linking uncertainty to semantically complete text spans may naturally bridge the two. The paper formally proposes Span-Level Uncertainty Estimation (SLUE) and provides initial method ( SpanUQ ) and benchmark ( SpanUQ-Bench ).

Many open questions remain:

How to improve uncertainty calibration so it becomes a trustworthy decision signal for downstream systems?

How to leverage span-level uncertainty for more efficient, targeted verification and self-refinement?

How to reliably transmit and interpret fine-grained uncertainty across agents with different base models?

Can span-level uncertainty extend beyond factual claims to complex reasoning, code generation, and other structured outputs?

As LLMs and multi-agent systems advance, uncertainty estimation's goal may expand from pure model evaluation to broader Uncertainty-Aware Decision Making , where semantically grounded uncertainty signals play an increasingly vital role. The authors hope SpanUQ serves not only as a method for SLUE but as a starting point for the community to explore fine-grained uncertainty estimation. Code, models, and benchmark are publicly released: https://arxiv.org/abs/2607.05721, https://damon-demon.github.io/SpanUQ.html, https://github.com/damon-demon/SpanUQ, https://huggingface.co/DamonDemon/SpanUQ, https://huggingface.co/datasets/DamonDemon/SpanUQ-Benchmark.