LLM Compression of Financial Texts Alters Decisions Despite Factual Accuracy

Abstract

Financial decision-makers face information volumes beyond direct review, making context compression indispensable. However, when LLMs compress financial source materials, they may alter the investment judgments those materials support. The paper defines this as Information Fidelity : compression loses fidelity when it changes the decisions induced by the source. In agentic systems, such losses can recur across intermediate steps and amplify through the decision pipeline.

The research team spans two financial corpora — 10-Q MD&A sections (N=300) and earnings call transcripts (N=297) from S&P 100 companies in FY2025 Q1–Q3 — and finds that LLM-based compression produces fluent, factually credible summaries that nevertheless change downstream decisions. Two diagnostic failure patterns are analyzed: Decontextualization (retaining headline evidence but stripping cautions, offsets, comparisons, and forward-looking qualifications needed for correct interpretation) and Model Dependency (different compressors expose systematically different views of the same source). On this basis, the paper proposes Agentic Context Compression (ACC) , which generates multiple candidate compressions and audits their divergences against the source. Results show financial text compression should be evaluated not only for efficiency or factuality but for its ability to preserve decision-relevant context.

Background

LLMs are increasingly used to compress long documents into shorter contexts for downstream models or human decision-makers. In high-stakes domains like finance, compressed text is valuable only if it retains the source evidence needed to support decisions. Long-context compression is inherently open-ended: a financial source text has no single correct summary because different summaries can select different evidence while remaining fluent and credible. The paper studies a simple but consequential failure mode: LLM-compressed representations can lead decision-makers to make different financial judgments than the full source would support. This problem goes beyond hallucination or factual inconsistency — it depends on which evidence stays visible, how it is contextualized, and how competing signals are balanced.

Existing compression evaluations typically measure reconstruction quality, factual consistency, or surface-level information retention, but do not directly test whether compression preserves the decisions the source supports. Prior work shows LLM-generated summaries can shift reader decisions while remaining fluent and credible; scientific paper summaries often omit scope-limiting details and over-generalize conclusions. In finance, company names, asset classes, and prior knowledge systematically bias LLM investment judgments. This paper extends those insights to raw financial text, asking: which decision-relevant evidence is retained?

Problem Formulation

Information fidelity is defined as decision retention relative to the source : compression has high fidelity when the decision model's beliefs induced by the compressed text stay close to those induced by the source. Given source text s , compression model M produces compressed text c (measured in bullet points) subject to a fixed budget B : c = M(s) subject to |c| ≤ B. Budget B is fixed across methods so fidelity differences reflect what is retained, not how much is written. Decision model E maps any text to a belief distribution over labels V={bear, neutral, bull}. For each text, the average of R=3 independent decision runs yields source belief p_s and compressed belief p_c :

p_s = (1/R) · Σ_{r=1}^R E^{(r)}(s),  p_c = (1/R) · Σ_{r=1}^R E^{(r)}(c)

Two metrics quantify fidelity loss. Decision Flip is the fraction of source documents where the top decision changes after compression: Flip = (1/N) · Σ_{i=1}^N 1[v̂_c(i) ≠ v̂_s(i)]. Total Variation Distance (TVD) d_TV(p_c, p_s) measures belief distribution shifts even when the top decision does not change: d_TV(p_c, p_s) = (1/2) · Σ_{v∈V} |p_c(v) − p_s(v)|. Flip captures outright decision changes; TVD captures smaller belief shifts that may not cross the decision boundary.

Methods

4.1 Naive Prompt and Contextualization

Naive Prompt is a single-pass compression c = LLM(s; θ, P_comp): the model receives the full source and a fixed bullet budget, selecting what it deems most important. Contextualization uses the same fixed budget but changes the criterion for "worth retaining." Instead of selecting information that looks important in isolation, it asks the compressor to retain details that make each key point interpretable — relevant comparisons, cautions, offsetting signals, or qualifiers. It uses prompt P_ctx to generate a candidate context c̃ = LLM(s; θ, P_ctx). This also serves as a stopping condition: if the candidate spends budget on isolated facts while missing details needed for judgment, the compressor returns to the source to re-select. The mechanism is unchanged; only the selection criterion shifts from isolated salience to interpretable evidence.

4.2 Agentic Context Compression (ACC)

Agentic Context Compression (ACC) instantiates the same operator M as a two-stage agentic compression pipeline. Stage 1 — Candidate Generation: The agent invokes an LLM candidate-generation tool to produce contextualized candidate contexts c_k = LLM(s; θ_k, P_ctx) from the same source s using different compression models θ_k, where k indexes distinct compression models. Stage 2 — Divergence Audit: The agent audits the candidate set {c_k} for divergences against the source. At step t , it emits an action from the latest tool observation o_t and history h_{t−1} = (o_1, a_1, …, o_{t−1}, a_{t−1}): a_t = LLM(o_t, h_{t−1}; θ, P_agent). For claims or omissions that diverge across candidates c_k, the action can grep the source for relevant terms, then read only short spans near matching segments. The agent uses these spans to verify whether a candidate c_k over-claims relative to the source — e.g., making evidence appear more bullish, more bearish, or more certain than the source supports — then scores each c_k by over-claim risk and source coverage, finally submitting the full compressed context best supported by the source. ACC bridges multi-agent factuality verification and financial atomic-claim provenance, but targets decision-changing omissions and over-claims rather than factuality alone.

Experiments

5.1 Experimental Setup

Datasets: Main panel covers S&P 100 companies FY2025 Q1–Q3 across two source types — 10-Q MD&A sections (N=300), collected via EDGAR-Crawler from structured SEC filings; quarterly earnings call transcripts (N=297). Decision Model: Decision model E is Gemini-3.1-Flash-Lite. Given source s or compression c , it produces belief distributions for computing p_s or p_c under the same decision prompt. Decision model reliability was validated: three independent reads of the same raw source; MD&A self-consistency 83.7%, Fleiss' κ=0.743; earnings calls self-consistency 85.0%, Fleiss' κ=0.720, in the "substantial agreement" range. Baselines: Multiple compression baselines compared under the same budget. Chunking splits s into fixed-size chunks, compresses each separately, and concatenates. Token Pruning uses LLMLingua and its long-context extension LongLLMLingua, with Finance-Llama3-8B as the token scorer. Integrator obtains compressions from multiple models and merges them into a final context.

5.2 Main Results: Compression Causes Fidelity Loss

Compression changes the decision model's outputs beyond re-read noise. The decision model's stochasticity baseline (same model re-reading raw source, no compression): MD&A flip 11.0%, earnings calls flip 8.8%. All four single-pass compressors exceed this baseline by more than 2× on both source types — the excess is the compression-induced component.

MD&A (10-Q) core results: No-compression baseline flip 11.0%, TVD 0.078; GPT-5.4-Mini Naive Prompt flip 33.0%, TVD 0.172; Chunking flip 37.0%, TVD 0.155; Contextualization flip 21.3%, TVD 0.099; LLMLingua token pruning flip 53.0%, TVD 0.231; LongLLMLingua flip 50.0%, TVD 0.226; Multi-LLM Integrator flip 24.7%, TVD 0.128; ACC flip 20.3%, TVD 0.102 (best) .

Earnings calls core results: No-compression baseline flip 8.8%, TVD 0.058; GPT-5.4-Mini Naive Prompt flip 23.9%, TVD 0.137; Chunking flip 31.6%, TVD 0.147; Contextualization flip 22.2%, TVD 0.116; LLMLingua flip 50.8%, TVD 0.247; LongLLMLingua flip 39.1%, TVD 0.196; Multi-LLM Integrator flip 22.9%, TVD 0.128; ACC flip 18.5%, TVD 0.112 (best) .

Token-pruning methods (LLMLingua, LongLLMLingua) perform worst, with flip rates above 50%, showing that mechanically discarding low-information tokens severely damages decision-relevant evidence. ACC achieves the lowest flip rates and near-lowest TVD on both source types.

5.3 Model Dependency

Different compressors show systematic differences in both direction and magnitude. The paper plots per-ticker source-relative movements for each compressor: x-axis direction (bear to bull), y-axis TVD magnitude. If compression were model-invariant, panels would look similar; instead compressors occupy different regions — some skew bearish, some bullish, and high-TVD points differ by panel. This indicates compression-induced decision changes are not just information loss; single-model summaries may carry model-specific slant . The Integrator provides a simple check: merely merging a second model's summary reduces MD&A flip from 33.0% to 24.7%. ACC goes further by invoking two models and checking divergences against the source to reduce flips further.

5.4 Decontextualization

The paper diagnoses decontextualization with an offline fact checklist, referencing FActScore and SAFE. Each source disclosure is decomposed into atomic facts, linked to source sentences, and labeled as three types: headline (carries the primary investment signal), context (calibrates the signal through cautions, offsets, comparisons, or expectation framing), and boilerplate (background or procedural text). Diagnosis finds: under fixed budget, compression typically retains headline outcomes but drops the cautions, offsets, comparisons, and forward-looking qualifiers needed to interpret them. In MD&A compression, context share drops from 25% in source to 9% in single-pass compressions — this disproportionate loss makes factually accurate summaries read as less well-calibrated than their source. Earnings calls, being shorter and fitting more facts under the same budget, retain context better; the pattern is milder.

Add-back Diagnostic: Starting from single-pass summaries that flipped the decision, omitted facts are appended and the model re-reads. For MD&A flip cases, restoring dropped context facts recovers the source decision in 37% of flips, while boilerplate or random facts achieve only 16–19%. Earnings calls show the same ranking but smaller gaps: context 33%, placebos 19–27%. Contextualization's gains are largest where decontextualization diagnosis is strongest: MD&A flip drops from 33.0% to 21.3%; earnings calls, where context is already better retained, show smaller gain from 23.9% to 22.2%.

5.5 Industry Case Study

To test whether fidelity effects transfer to a production-grade forecasting system, the team applied ACC in a globally deployed commercial equity forecasting product for institutional investors, based on S&P 100 FY2025 Q1–Q3 disclosures. Absolute product metrics are omitted for commercial confidentiality. Relative to raw disclosures, both naive compression and ACC reduce token consumption by over 90%. However, naive compression reduces predictive IC (Information Coefficient), indicating compression-induced decision changes remove forecast-relevant disclosure signals. By contrast, ACC improves predictive IC by 8.3% relative to the raw-source baseline and by 23.8% relative to naive compression, while reducing source-relative decision flips by 59.5%. This result suggests preserving decision-relevant context matters not only for compression quality but for retaining downstream predictive signal in real financial systems.