How SAP Cuts 90% Compute and Boosts 4K Panorama Segmentation Accuracy by 17.2%

Why SAM2 Fails on 4K Panoramas

Applying SAM2 directly to a 4K equirectangular panorama forces the image into a 1024×1024 input, wasting over 90% of pixels as black padding and severely distorting geometry at the poles and seam, which breaks the model's topological continuity and its Stream memory mechanism that expects smooth, temporally ordered video frames.

SAP: Turning a Panorama into a Pseudo‑Video

SAP’s key insight is to align the panorama’s topology with SAM2’s perspective‑video prior by rendering overlapping perspective views along a carefully designed camera trajectory, effectively creating a “pseudo‑video” that the model can process frame‑by‑frame.

Scanning Trajectory Design

The authors define a grid on the sphere and adopt a column‑first zig‑zag path ("col‑first") with 90° field‑of‑view and 50% overlap. This yields an infinite‑loop property: the last frame differs from the first only in yaw, allowing seamless looping and diverse training samples.

Each camera pose projects the user’s click prompt onto every frame by converting the spherical direction to camera coordinates and then to pixel coordinates, ensuring consistent prompts across frames.

Fine‑Tuning SAM2 for the Pseudo‑Video

Freeze Backbone : The Hiera‑Large encoder is frozen; only the memory encoder, attention, mask decoder, and prompt encoder are updated.

Mixed‑Data Training : To avoid catastrophic forgetting, original SAM2 data (SA‑1B images and SA‑V videos) are mixed with the synthetic panorama data.

Large‑Scale Synthetic Data : Using the InfiniGen engine, 183 k 4K panoramas with >6.4 M instance masks are generated to provide abundant supervision.

Experimental Validation

Zero‑shot tests on three challenging 4K/8K benchmarks show consistent gains.

SOTA Comparison on Real‑World 4K Panorama (PAV‑SOD)

Original SAM2‑Large: mIoU 58.3

SAM2‑SCAN (only scanning): mIoU 67.8 (+9.5)

Full SAP‑Large: mIoU 75.5 (+17.2 over baseline)

Average improvement across model sizes is +17.2 mIoU.

Generative 8K Panorama (HunyuanWorld‑1.0)

Despite style and resolution shift, SAP maintains a clear advantage, demonstrating strong domain generalisation.

Synthetic 4K Panorama (InfiniGen)

Under a “train‑same, test‑unseen” setting, SAP again outperforms SAM2‑SCAN and the original model, confirming the robustness of the pseudo‑video approach.

Ablation Studies

Data & Training Strategy : Training only on synthetic data drops performance to 54.0 mIoU, while mixed data yields the best results.

Scanning & Overlap : No scanning → 62.1; scanning without overlap → 69.8; scanning with overlap → 75.5 (best).

Perspective Transform vs. Direct ERP Patching : Direct ERP patches achieve 70.2 mIoU, far below the perspective‑scan pipeline’s 75.5, confirming the necessity of aligning input to SAM2’s perspective prior.

Qualitative Results

Visual comparisons on PAV‑SOD, HunyuanWorld‑1.0 and various indoor/outdoor scenes show SAP producing seamless, complete masks, while SAM2 suffers from seam breaks and missing objects.

Advantages, Limitations, and Future Directions

Fundamental Innovation : Introduces the “topology‑memory alignment” concept.

Performance Leap : Large zero‑shot gains on 4K/8K panoramas.

Practical Pipeline : End‑to‑end data synthesis, training, and inference workflow.

Inspiration : The idea of converting static geometry into a temporal signal can benefit other high‑resolution or non‑Euclidean data.

Limitations include the still‑significant compute required to render dozens of 1024×1024 frames, dependence on SAM2 as a plug‑in, and the current focus on static panoramas rather than full‑panorama video.

Future work may explore native spherical models and extending the paradigm to panoramic video segmentation.

Reference

SAP: Segment Any 4K Panorama