Tagged articles
Data Compression
66 articles · Page 1 of 1
This comprehensive tutorial explains time-series data fundamentals, compares InfluxDB with MySQL and other popular TSDBs, provides step-by-step Linux installation commands for InfluxDB 2.7, and demonstrates basic write and query operations using the command-line interface.
Data CompressionData RetentionInfluxDB
0 likes · 14 min read

AliSQL integrates DuckDB as its storage engine to achieve high‑density columnar compression and fast analytical scans, detailing DuckDB’s multi‑layer storage format, adaptive compression algorithm selection, performance benchmarks versus InnoDB, HBase, ClickHouse, OceanBase, and the engineering optimizations AliSQL adds for throughput and cost reduction.
AliSQLData CompressionDuckDB
0 likes · 12 min read

This article explores ClickHouse’s high‑performance OLAP architecture, covering its MPP design, columnar storage, vectorized execution, pre‑sorting, table engines, data types, sharding and replication strategies, as well as index designs that together enable rapid analysis of massive datasets.
ClickHouseData Compressioncolumnar storage
0 likes · 15 min read

Gaode’s Uninhabited Area project introduces Satellite Rescue, which transmits encrypted, natural‑language emergency messages via satellite and a rescue chatroom to boost response speed, and Offline Navigation, employing quadtree indexing, predictive caching and seamless mode switching to guide users through network‑dead zones, proven in harsh Inner Mongolian terrains.
Data CompressionMobile AppSatellite Communication
0 likes · 8 min read

This article explains how large language models act like a super‑library by compressing vast amounts of text using information‑theoretic concepts, probability‑based coding, autoregressive neural networks, and arithmetic coding, while discussing accuracy, compression ratios, and theoretical limits.
AIData Compressionarithmetic coding
0 likes · 8 min read

This article explains how a CDP system tackles the storage and set‑operation challenges of billions of user‑ID tags and groups by adopting bitmap techniques, especially RoaringBitmap, to dramatically reduce space usage and enable fast union, intersection, and difference calculations.
Data CompressionJavaRoaringBitmap
0 likes · 9 min read

vivo migrated five 20‑TB MySQL clusters to OceanBase using OCP, oblogproxy, and OMS, eliminating sharding costs, achieving over 70% storage savings, improving consistency and performance, and leveraging native distributed architecture, tenant isolation, and strong compression to support scalable, reliable operations.
Data CompressionDatabase MigrationDistributed Database
0 likes · 16 min read

This article explains why bit manipulation is a staple technique in computer science, detailing performance gains, state flag encoding, mask operations, data compression, cryptography, image processing, hash structures, binary counting, and algorithm design, with practical Go examples.
Data CompressionGobit-manipulation
0 likes · 3 min read

This article explores why 1980s games like Contra could deliver rich graphics, music, and gameplay within just 128KB by using tile-based graphics, specialized audio chips, and ultra‑compact code, contrasting those techniques with modern developers' expectations of massive storage.
Data CompressionRetro GamingTile Graphics
0 likes · 7 min read

In 2023, a TapTap infra team reduced Redis operating costs by 460,000 CNY per month through low‑cost ESSD instances, traffic compression, unused‑instance cleanup, TTL management, data migration, online compression, and targeted cleaning, detailing ten concrete measures and the open‑source tools that enabled zero‑downtime optimization.
Cost OptimizationData CompressionInfrastructure
0 likes · 17 min read

The article reviews core linear algebra concepts—vectors, matrices, eigenvalues, and transformations—and demonstrates how PaddlePaddle’s paddle.linalg API enables practical tasks such as least‑squares regression, image compression via SVD, PCA‑based dimensionality reduction, and broader machine‑learning, graphics, cryptography, and optimization applications.
Data CompressionPCAPaddlePaddle
0 likes · 10 min read

The article explains BitMap fundamentals and introduces Roaring BitMap’s compressed container architecture—Array, BitMap, and Run containers—detailing their conversion logic, Java implementation snippets, performance advantages over traditional BitSets, and practical API usage for high‑performance, memory‑efficient big‑data applications.
Data CompressionJavaRoaring Bitmap
0 likes · 18 min read

The article presents a sparse‑matrix compression scheme called smdbz for high‑resolution radar dBZ data, detailing the dBZ grade encoding, CSR representation, custom Python implementation, bit‑packing techniques, file layout, performance comparisons with PNG and NPY formats, and the method's advantages and limitations.
CSRData CompressionPython
0 likes · 21 min read

WCDB 2 introduces multi-language support (C++, Java, Kotlin), a rewritten SQL builder, fast backup and recovery with sub‑second times, incremental WAL‑based backup, automatic data migration, Zstandard compression, on‑the‑fly column addition, and performance enhancements such as an FTS5 optimizer and interruptible transactions, delivering a robust high‑performance mobile database solution.
Data CompressionSQL BuilderSQLite
0 likes · 47 min read

The article explains how early 1980s games such as Contra managed to deliver extensive graphics, music, and gameplay within a mere 128KB by using tile‑based graphics, specialized audio chips, and highly optimized assembly code, highlighting the stark contrast with modern app sizes.
Data CompressionRetro GamingTile Graphics
0 likes · 6 min read

This article explores why 1980s games like Contra could deliver rich graphics, music, and gameplay within just 128KB by examining data size perception, tile‑based graphics, limited audio chips, and ultra‑compact code techniques that contrast sharply with modern app bloat.
Data CompressionRetro GamingTile Graphics
0 likes · 7 min read

Trace2.0 is an OpenTelemetry‑based application monitoring system that processes petabyte‑scale trace data using multi‑channel client protocols, gRPC, load‑balancing optimizations, ZSTD compression, Kafka pipelines, ClickHouse storage, and a JDK 21 upgrade with virtual threads, achieving significant performance and cost improvements.
ClickHouseData CompressionJDK21
0 likes · 15 min read

This article explains the motivation, dictionary design, bigram table, encoding rules, and real‑world compression results of the Smaz2 library, a space‑optimized C/Python compressor for short LoRa messages on microcontrollers with less than 2 KB RAM.
C++Data CompressionLoRA
0 likes · 8 min read

This article details a courier company's evolution from a simple file‑based logging system to a cloud‑native log center, describing the limitations of the original architecture, the migration to an ELK stack, subsequent challenges, and the final redesign using ClickHouse for high compression, low cost, and improved query performance.
ClickHouseData Compressionelk
0 likes · 12 min read

This article presents a systematic, nine‑grid method for reducing MySQL storage expenses—including table compression, JSON field serialization, and hot‑cold data separation—while quantifying benefits, ensuring data safety, and validating system stability through staged testing and SRE metrics.
Data CompressionDatabase Cost ReductionMySQL
0 likes · 13 min read

This article presents a comprehensive, step‑by‑step methodology for cutting MySQL storage expenses—including background analysis, challenge identification, a nine‑grid systematic framework, benefit calculation, safety and stability verification, rollout, gray‑release, and rollback—demonstrating over 50% disk space reduction while managing risk.
Data CompressionMySQLStorage Management
0 likes · 14 min read

This article describes how China Unicom’s Western Innovation Research Institute designed and deployed a centralized, real‑time log analytics platform using Apache Doris, detailing the migration from Hive and ClickHouse, performance optimizations, storage cost reductions, and the resulting improvements in data ingestion, query speed, and operational efficiency.
Apache DorisCold‑Hot Data ManagementData Compression
0 likes · 18 min read

This article examines the massive data storage and real‑time analysis needs of an activity platform, evaluates MySQL, sharded MySQL, Elasticsearch and HBase, and explains why ClickHouse—with its columnar storage, MergeTree engine, vectorized execution, and distributed architecture—offers the best balance of write performance, query speed, and scalability for billions of records.
ClickHouseColumnar DatabaseData Compression
0 likes · 31 min read

This article explains how column store indexes in SQL Server store each column separately, dramatically improve query performance through batch processing and compression, outlines their physical structure, encoding methods, creation syntax, maintenance steps, and space usage considerations.
Column StoreData CompressionSQL Server
0 likes · 11 min read

This article recounts how a MIT information‑theory class assignment sparked David Huffman's invention of a bottom‑up binary‑tree coding method that revolutionized lossless data compression and became a cornerstone of modern computer science.
AlgorithmData CompressionHuffman coding
0 likes · 9 min read

This article explains how early 1980s games like Contra fit extensive storylines, graphics, and music into just 128KB by using tile‑based graphics, minimal audio data, and highly optimized code, illustrating the dramatic shift in data expectations as technology advances.
Data CompressionRetro GamingTile Graphics
0 likes · 6 min read

This article details comprehensive strategies for handling massive seat‑selection data on mobile devices, covering data modeling, interface pre‑loading, static and dynamic compression, view hierarchy tweaks, bitmap reuse, drawing optimizations, hardware acceleration, and multithreaded task handling to dramatically improve load times and frame rates.
AndroidCachingData Compression
0 likes · 21 min read

Client‑side developers can boost API speed and user experience by collaborating with back‑end teams to cut server processing and network latency, using caching, concurrency, smaller or UDP‑based packets, and aggressively pruning or compressing payloads, which can shrink megabyte responses to tens of kilobytes and shave dozens of milliseconds off latency.
API optimizationCachingData Compression
0 likes · 9 min read

This article explains the fundamental differences between row‑based and column‑based storage in databases, compares their write and read performance, lists pros and cons, presents an openGauss experiment with sample tables, and offers practical guidance on when to choose each storage model.
Column StorageData CompressionDatabase Performance
0 likes · 9 min read

This article summarizes recent Chinese database product strategy reports, outlining database definitions, management systems, design processes, product classifications, architectural layers, HTAP technology, compression methods, storage index structures, intelligent autonomous optimization, and deployment models, highlighting trends and future directions in the database industry.
Data CompressionDatabase ArchitectureHTAP
0 likes · 8 min read

The inaugural Ship‑Sea Data Intelligent Application Innovation Competition, co‑hosted by Taihu Lab, Huawei and local authorities, challenged participants to compress massive unstructured marine data, and the winning teams revealed novel preprocessing and encoding‑compression pipelines that dramatically improve storage efficiency while preserving data integrity.
AIData Compressionmarine technology
0 likes · 7 min read

This article details Qunar's high‑concurrency flight pricing architecture, covering the origin of pricing data, two‑level caching design, a cache‑management system, CQRS‑based policy storage, data compression techniques, and performance improvements achieved through these optimizations.
CachingData Compressionbackend architecture
0 likes · 20 min read

This article chronicles Jacob Ziv’s pioneering contributions to lossless data compression, tracing the evolution from early Morse code through Shannon‑Fano and Huffman to the groundbreaking LZ77 and LZ78 algorithms, and highlights his biography, major awards, and lasting influence on modern digital media and storage technologies.
Data CompressionJacob ZivLZ77
0 likes · 9 min read

The article chronicles Jacob Ziv's life and achievements, detailing the development of the LZ77 and LZ78 lossless compression algorithms, their historical context, and the lasting impact of these innovations on modern data storage and transmission technologies.
Data CompressionJacob ZivLempel-Ziv
0 likes · 8 min read


21CTOSep 18, 2021 · Fundamentals
This article introduces ten foundational algorithms—from sorting and Fourier transforms to encryption and graph analysis—explaining their core principles, historical origins, and the pivotal roles they play in today’s software, security, and data processing systems.
AlgorithmsData CompressionSignal Processing
0 likes · 9 min read

This article provides a comprehensive introduction to Apache Parquet, covering its origins, columnar storage advantages, nested schema support, internal architecture, storage model components, comparison with ORC, and practical tools for inspecting Parquet files.
Data CompressionHadoopORC Comparison
0 likes · 10 min read


21CTOJul 24, 2021 · Fundamentals
Jacob Ziv, the 90‑year‑old Israeli pioneer honored with the 2021 IEEE Medal of Honor, co‑created the groundbreaking Lempel‑Ziv lossless compression algorithms that underpin modern formats like GIF, PNG and ZIP, and his century‑spanning career reshaped data storage, transmission, and information theory.
Data CompressionJacob ZivLempel-Ziv
0 likes · 10 min read

This article explains how to reduce memory usage of massive Java int/long arrays by applying real‑time compression, eliminating redundancy, using indexed buckets, offset storage, and a series of low‑level optimizations that boost TPS from dozens to over a thousand while preserving random‑access capabilities.
Data Compressionarraymemory optimization
0 likes · 14 min read

This article explains the principles of data compression, contrasting lossy and lossless methods, introduces common algorithms such as Run‑Length Encoding, dictionary coding, and Huffman coding, and discusses extreme cases like compression bombs, illustrating concepts with examples and visual diagrams.
Data CompressionHuffman codingcompression algorithms
0 likes · 9 min read

This article details how Ctrip's senior engineering manager leveraged ClickHouse to build a high‑availability, sub‑second response data platform handling nearly 700 billion rows, describing the motivations, architecture, data synchronization processes, performance gains, challenges, and practical recommendations for large‑scale analytics.
ClickHouseData Compressionbig data
0 likes · 28 min read


ITPUBNov 12, 2020 · Databases
This article details how Vivo's cloud service tackled explosive data growth by applying horizontal and vertical sharding, routing‑table based dynamic expansion, and MySQL InnoDB compression, providing step‑by‑step guidance, performance results, and practical recommendations for large‑scale database deployments.
Data CompressionInnoDBMySQL
0 likes · 16 min read

Vivo Cloud Service scaled from millions to tens of millions of users by applying four sharding strategies—horizontal table partitioning, horizontal database splitting, vertical partitioning, and dynamic routing—while enabling InnoDB compression, which halved storage size, kept CPU impact modest, and preserved ample free capacity for future growth.
Data CompressionInnoDBMySQL
0 likes · 13 min read

ClickHouse is a PB‑scale, open‑source columnar OLAP database that uses a ZooKeeper‑coordinated sharded cluster, columnar storage, vectorized execution, advanced compression, data‑skipping indexes, and materialized views to deliver high‑performance interactive analytics, yet it requires manual shard management, lacks a mature MPP optimizer, and handles real‑time single‑row writes poorly.
ClickHouseData CompressionOLAP
0 likes · 18 min read


ITPUBOct 12, 2020 · Databases
This article explains how ClickHouse’s column‑oriented storage, vectorized execution engine, rich DBMS capabilities, flexible table engines, and carefully designed distributed architecture enable it to handle massive workloads with sub‑second query latency, making it a standout OLAP solution.
ClickHouseColumnar DatabaseData Compression
0 likes · 29 min read

This article explains the significance of data compression, outlines when it should be applied, compares lossless algorithms, discusses segment selection, and details how Kafka and RocketMQ implement message compression to improve throughput while balancing CPU, storage, and network resources.
Algorithm SelectionData CompressionKafka
0 likes · 9 min read

This article provides a comprehensive technical overview of ClickHouse, covering its complete DBMS capabilities, column‑oriented storage and compression, vectorized execution engine, relational SQL support, diverse table engines, multi‑master clustering, sharding, and the design philosophies that make it exceptionally fast for large‑scale analytical workloads.
ClickHouseColumnar DatabaseData Compression
0 likes · 29 min read

This article explains what an algorithm is, outlines its essential properties, traces its historical roots, and introduces ten pivotal algorithms—from sorting and Fourier transforms to RSA, hash functions, and random number generators—that underpin today’s digital world.
AlgorithmsData CompressionNetworking
0 likes · 10 min read

To combat Foreground Out‑Of‑Memory (FOOM) crashes in iOS, we describe the evolution from a simple FBAllocationTracker‑based logger to a high‑performance monitoring system that collects, stores, compresses, reports, and visualizes allocation data using custom hooks, splay trees, hash‑based stack compression, and selective reporting.
Data CompressionFOOMiOS
0 likes · 16 min read

This article presents a comprehensive overview of Alibaba's Time Series Database (TSDB), covering its business panorama, covered scenarios, challenges, performance evolution, architecture, core compression and query technologies, stability mechanisms, and future development plans.
Alibaba CloudData CompressionHigh Frequency Queries
0 likes · 11 min read

The IBM FlashSystem 9100 Q&A details its NVMe‑based all‑flash architecture, performance advantages over V9000, unique hardware compression, redundancy, QoS, cloud connectivity via S3 and OpenStack, and use‑case scenarios for high‑throughput, multi‑cloud enterprise environments.
Data CompressionFlashSystem9100IBM
0 likes · 11 min read

This article recounts a 2017 internal high‑performance competition where a C server handling a 40 million‑row user‑info dataset was progressively optimized through data compression, custom hash tables, memory layout redesign, and a thread‑pool model, ultimately achieving a four‑fold throughput increase.
C++Data Compressionhash table
0 likes · 13 min read

By serializing user feature data with Java's Protostuff (built on Protobuf) and then applying JDK Gzip compression before storing it in Redis, the author shrank typical 70 KB per‑user payloads to under 10 KB, enabling billions of records with cross‑language compatibility and no schema‑breakage.
Data CompressionGZIPJava
0 likes · 6 min read

Using Meituan’s hotel‑travel platform as a case study, the article details a scalable online feature system architecture that combines layered storage, efficient compression, and robust synchronization to meet extreme concurrency, throughput, terabyte‑scale data, and sub‑10 ms latency demands for AI‑driven strategy services.
Data Compressionbig datadistributed storage
0 likes · 23 min read

Apache CarbonData 1.0.0, now an Apache incubating project, adds over 80 new features and bug fixes—including a new data loading solution, Spark 2.1 integration, update/delete SQL support, adaptive compression for numeric types, B‑Tree LRU cache, V2 format for faster first‑query performance, vectorized reader, bucket‑table joins, off‑heap memory, single‑pass loading, and pre‑generated dictionaries—aimed at delivering faster, more flexible, and efficient columnar storage for big‑data workloads.
Apache CarbonDataData CompressionSpark Integration
0 likes · 8 min read

Alibaba's HiStore columnar database, deployed for e‑commerce historical data, processes over 6 trillion records and 5 PB of data daily, offering high compression, low cost, linear scalability, MySQL compatibility, and superior OLAP performance for massive multi‑dimensional queries.
AlibabaColumnar DatabaseData Compression
0 likes · 7 min read

Alibaba’s HiStore column‑store database handles over 6 trillion records and 5 PB of data daily, delivering ultra‑high compression, low hardware cost, and millisecond‑level multidimensional queries for massive e‑commerce and cloud workloads.
AlibabaColumnar DatabaseData Compression
0 likes · 7 min read

HiStore is a columnar database developed by Alibaba's middleware team, designed for massive OLAP workloads with high compression ratios, low storage and maintenance costs, supporting ad‑hoc multi‑dimensional queries, knowledge‑grid optimization, efficient data loading, and offering features such as approximate queries and high‑availability clustering.
Columnar DatabaseData CompressionOLAP
0 likes · 19 min read

HiStore, an Alibaba middleware team's columnar database, offers high‑compression, low‑cost OLAP storage for massive datasets, leveraging a knowledge‑grid engine, column‑based storage, and efficient compression to achieve multi‑dimensional ad‑hoc queries with performance gains of up to tens of times over traditional row‑based systems.
Data CompressionHiStoreOLAP
0 likes · 17 min read


21CTOJun 18, 2016 · Databases
HiStore, Alibaba’s columnar database built on a patented Knowledge‑Grid, delivers ultra‑high compression (over 10:1, up to 40:1), low‑cost storage, rapid query performance, linear scalability, and seamless MySQL compatibility, making it ideal for massive OLAP workloads and real‑time analytics across diverse industries.
Columnar DatabaseData CompressionOLAP
0 likes · 8 min read

This article explains Apache Parquet’s columnar storage format, its support for nested data models, the underlying striping/assembly algorithm, file structure, push‑down optimizations, and performance advantages within the Hadoop ecosystem, providing a comprehensive guide for big‑data practitioners.
Apache ParquetData CompressionHadoop
0 likes · 22 min read

This article explains how Kafka’s design—using sequential disk reads/writes, zero‑copy system calls, file segmentation, batch sending, and message compression—delivers massive throughput while minimizing performance loss and network load.
Data CompressionDistributed Messagingbatch sending
0 likes · 4 min read

This article explains data deduplication technology, its definition, various classification schemes based on execution time, block size, granularity, and location, and compares it with data compression, highlighting how both techniques can be combined to maximize storage savings.
Data Compressionbackuphashing
0 likes · 6 min read


21CTOOct 21, 2015 · Mobile Development
This article examines why mobile app log reporting consumes excessive traffic and presents concrete methods—such as using concise key‑value URLs, batching uploads, shortening domains and keys, and compressing data—to dramatically reduce bandwidth usage while preserving data timeliness.
Data CompressionHTTPMobile App
0 likes · 7 min read

Facebook’s data warehouse, storing over 300 PB and growing by 600 TB daily, transitioned from the RCFile format to an optimized ORCFile implementation, achieving 5‑8× better compression and up to three‑fold faster write performance while maintaining high read efficiency.
Data CompressionFacebookHive
0 likes · 14 min read

This article surveys a collection of seminal algorithms—including Huffman coding, public‑key encryption, Dijkstra's shortest‑path, binary search, quicksort, Karatsuba multiplication, Euclid's GCD, Bresenham's line drawing, and the fast inverse square‑root—explaining their origins, principles, and lasting impact on computer science.
AlgorithmsData Compressioncomputer graphics
0 likes · 7 min read

Learn how to harness multi‑core CPUs to dramatically speed up common Linux utilities such as grep, bzip2, awk, sed, and wc by using GNU Parallel with the –pipe (or –spreadstdin) option, turning single‑threaded commands into efficient parallel map‑reduce pipelines for massive data files.
Data CompressionGNU ParallelLinux
0 likes · 5 min read
