Apple's LensVLM: Compress Long Text to Images, Zoom In for Details

Problem: Long-Context Accumulation in Agents

When agents execute long-horizon tasks, code, tool outputs, and interaction records accumulate, causing context length to grow. Summarization or compression can reduce input but risks discarding details that become critical later. If the model only sees compressed output, it cannot retrieve the original information.

Background: Text-as-Image Compression

DeepSeek-OCR demonstrated that rendering text as images can represent textual information with fewer visual tokens than text tokens. A compressed page can cover the full document, but readability becomes a bottleneck: smaller characters are harder for the vision encoder to recognize. At 5× compression, a base model reading compressed images directly achieved only 31.3% average accuracy, far below the full-text baseline of 72.4%.

LensVLM Approach: Browse Then Magnify

Apple's Foundation Models team and Duke University propose LensVLM. The model first receives the question and all compressed pages (covering the entire document), selects pages worth reading in detail, then invokes an expand tool . For text-rendered pages, the tool returns the original text and adds it to the context. The model can continue selecting pages until evidence is sufficient.

In a demo, given 15 compressed pages and the question "What government position did the actress who played Corliss Archer in Kiss and Tell hold?", the model selected page 10, called read_page to retrieve the original text, and answered "Chief of Protocol of the United States".

Training: Supervised Fine-Tuning Then Reinforcement Learning

The base model is Qwen3.5-9B-Base. Researchers first filtered samples the base model could not answer correctly, then used Qwen3.5-397B to generate tool-use trajectories conditioned on the question, ground-truth evidence, and reference answer. Supervised fine-tuning (SFT) taught the model reasoning, page selection, and answering. Subsequently, DAPO reinforcement learning optimized the model's own trajectories, rewarding correct answers and giving extra credit for tool use only when the answer was correct.

At 5× initial compression, SFT raised average accuracy from 31.3% to 63.7%, and RL further improved it to 68.9%, approaching the full-text baseline of 72.4%.

Evaluation Across Compression Levels

Main benchmarks: NQ, HotpotQA, MuSiQue, HELMET, plus out-of-training RULER, Qasper, and LongBench, covering single-hop and multi-hop long-document QA.

At 15× initial compression, base model accuracy dropped to 18.1%; LensVLM achieved 52.1%.

Effective compression rate (original tokens / (compressed image tokens + expanded tokens)): 4.3× at 5×, 7.4× at 10×, 10.1× at 15×.

Speed and Cost Benefits

For 32K–140K token contexts at 15× initial compression:

End-to-end latency: full-text input 3.62 s, LensVLM 2.83 s, with offline pre-cached pages 2.52 s (~30% reduction).

Peak KV cache: ~1/14 of full-text input.

Saturated throughput: pre-cached version 3.7× higher.

Estimated GPU cost per 1K requests: ~73% reduction (at $6/GPU-hour).

Short contexts still favor direct full-text input; advantages appear for long documents and repeated queries over the same cached pages.

Handling PDF Documents: OCR vs. High-Resolution Pages

For native PDFs containing tables, figures, and layout, two expansion variants were trained: one returns OCR text, the other returns the high-resolution page image. On MMLongBench-Doc at 15× pixel reduction:

Direct compressed page reading: 31.1%

OCR expansion: 32.1%

High-resolution page expansion: 41.9% at 10.3× effective compression.

Visual information (charts, layout) preserved in high-res pages helps more than OCR text, but incurs higher input cost. The expansion format should match content type: original text for pure-text pages, high-res images for visually rich documents.

Conclusion and Future Direction

LensVLM's core idea is to let the model browse compressed content while retaining on-demand access to the original. The model allocates its reading budget to pages relevant to the current question. As context windows grow, teaching models to actively manage reading budget—deciding what to skim, what to read deeply, and when to look back—may become as important as extending the window itself.

Resources

Paper: https://arxiv.org/abs/2605.07019

Model: https://huggingface.co/apple/LensVLM-9B

Code: https://github.com/apple-aiml-research/ml-lensvlm

Demo: https://huggingface.co/spaces/hugging-apps/lensvlm-9b-demo