Distributed Storage and Application Solutions for Massive Spatiotemporal Data
This article explains the rapid growth of global spatiotemporal data, the limitations of traditional GIS, and presents SuperMap's distributed storage architecture, unified data access APIs, dynamic rendering techniques, and geographic processing modeling with real‑world case studies to address performance and scalability challenges.
According to IDC, global data volume doubles every 18 months and is projected to reach 175ZB by 2025, with 80% related to spatial location, creating unprecedented challenges for traditional GIS in data management, analysis, and visualization.
To handle this surge, SuperMap proposes a new distributed storage approach that supports both static and streaming spatiotemporal data, ensuring high‑performance analysis and map rendering while remaining user‑friendly for GIS professionals.
The solution includes:
Classification of heterogeneous spatiotemporal data (vector, raster, file‑based) and recommendation of appropriate distributed databases such as relational DBMS, MongoDB, Elasticsearch, HDFS, and HBase.
A unified data‑read/write API that abstracts various data sources (HDFS engine, object storage engine, relational spatial DB engine, NoSQL engine) and allows source type identification via parameters.
On‑demand multi‑source queries that push filter conditions to the storage layer, reducing memory pressure compared to traditional SparkSQL approaches.
Four major application scenarios are demonstrated:
Large‑scale agricultural data ingestion and analysis, achieving county‑level processing in ~10 minutes, province‑level in ~30 minutes, and national‑level in ~2 hours.
Sichuan land‑rights analysis, reducing a 40‑minute overlay to ~2 minutes with a 4‑node cluster.
Global cropland statistics, completing a 400‑image raster analysis in 40 minutes on a 6‑node cluster (50× speedup).
High‑performance dynamic map rendering, enabling vector tile generation and real‑time density analysis for billions of points with orders‑of‑magnitude performance gains.
SuperMap also introduces a distributed geographic processing modeling platform that integrates distributed storage, computation, and visualization, offering over 600 predefined tools for tasks such as distributed data management, spatial analysis, and map service deployment.
The concluding section highlights how the integrated technologies solve the “two difficulties and two slownesses” of massive spatiotemporal data—reading difficulty, analysis slowness, visualization slowness, and distributed technology adoption—earning industry recognition and awards.
DataFunTalk
Dedicated to sharing and discussing big data and AI technology applications, aiming to empower a million data scientists. Regularly hosts live tech talks and curates articles on big data, recommendation/search algorithms, advertising algorithms, NLP, intelligent risk control, autonomous driving, and machine learning/deep learning.
How this landed with the community
Was this worth your time?
0 Comments
Thoughtful readers leave field notes, pushback, and hard-won operational detail here.