This guide explains why file compression matters for storage and bandwidth, then details the most common Linux compression utilities—including tar, gzip, lzma, xz, bzip2, pax, peazip, 7zip, shar, cpio, ar, iso, kgb, zpaq, and File Roller—along with installation steps, key options, and practical command examples.
This article explains why Kafka can handle up to 20 million messages per second and 600 MB/s throughput by detailing producer batching and custom protocols, broker page‑cache, file layout and zero‑copy techniques, as well as consumer group strategies for efficient message consumption.
This article analyzes the performance and storage problems of the Windows version of WeChat, explains why message growth causes slowness, large file size, and corruption risk, and presents a multi‑pronged solution involving sharding, indexing, payload compression, and robustness improvements that halve database size and boost I/O performance by about ten percent.
This article presents an in‑depth overview of time‑series data concepts and the practical adoption of TimescaleDB built on PostgreSQL at Hytera, covering requirements, core features, hypertable architecture, deployment models, compression, data retention, backup strategies, and performance benchmarks.
Tencent's TXAV1 encoder, a self‑developed AV1 video codec, combines advanced compression‑rate improvements, engineering acceleration, and extensive cloud deployment, achieving up to 78‑334% speedup with minimal quality loss, and includes novel pre‑analysis models, adaptive quantization, and multi‑threading strategies for both video and image applications.
AWS switched S3 log compression from gzip to Zstandard, saving roughly 30% of storage space—equivalent to exabytes—by changing the internal storage method, a move confirmed by employee comments and a coinciding 31% price drop in late 2021.
This article explains how Elasticsearch leverages inverted indexes, term dictionaries, and compression techniques such as Frame‑of‑Reference and Roaring Bitmaps to enable rapid full‑text search, detailing the underlying data structures, query processing, and practical indexing tips for efficient backend search implementations.
This article explains how Elasticsearch uses inverted indexes, term dictionaries, and advanced compression techniques such as Frame‑of‑Reference and Roaring Bitmaps to achieve rapid search performance while minimizing memory and disk usage, and it also covers practical indexing tips for production use.
This guide provides concise English explanations of dozens of essential Linux commands—including compression tools, process utilities, and network utilities—covering their purpose, syntax, key options, and practical usage examples for system administrators and developers.
The article examines a recent call to boycott 7‑Zip, outlining three alleged shortcomings—limited open‑source transparency, unresolved security vulnerabilities, and the developer's nationality—while presenting community rebuttals and suggesting alternative compression tools.
The article explains how ClickHouse uses block‑based batch processing combined with LSM‑style pre‑sorting and columnar compression to accelerate range queries on massive datasets, while also discussing the trade‑offs such as write latency and limitations for transactional workloads.
This guide explores systematic image optimization for web applications, covering formats like WebP, lossless vs lossy compression, lazy loading, placeholder strategies, format fallback, monitoring, and practical code implementations to reduce image size, memory usage, and improve loading speed and user experience.
This article details how a Java Spring Boot microservice's response time was reduced from 5‑6 seconds to under one second by applying gzip compression, parallel data fetching, short‑lived caching, MySQL index tuning, and JVM G1 garbage‑collector adjustments.
This article provides a comprehensive examination of InnoDB's physical file layout, detailing page components such as the file header, page header, record storage formats, variable‑length and NULL handling, free space, page directory, file trailer, external storage pages, and the differences among row formats and compression techniques, illustrated with concrete hexdump examples and SQL statements.
Video bitrate depends on content complexity, resolution, frame rate, and the chosen codec and bitrate‑control method, with modern encoders like HEVC and AV1 using advanced prediction, transform, quantization and entropy coding to dramatically reduce size while preserving perceived quality for smoother streaming.
This article explains why storing JSON strings in Redis is inefficient, compares JSON with binary‑safe formats like MessagePack, evaluates compression algorithms such as ZSTD and LZ4, and provides Python examples showing how to reduce latency and memory usage when using Redis strings.
This guide explains how to handle common compression formats—zip, tar.gz, rar, and 7z—in Python, covering built‑in modules, third‑party libraries, required installation steps, and complete code examples for both creating and extracting archives.
This article explains how to use OpenResty and Lua to integrate Redis caching, compress responses, schedule periodic updates, and configure URL‑based cache rules directly within Nginx, improving concurrency and resilience of web services.
This article explains how to identify gzip‑compressed responses that may be gzip bombs, how to inspect HTTP headers and raw response data using Python's requests library, and provides command‑line and code examples for measuring compressed and uncompressed sizes without triggering decompression.
This article provides a comprehensive overview of the most frequently used Linux compression utilities (zip, gzip, bzip2, xz) and the tar archiving command, detailing their syntax, key options, how to combine them, and practical examples for compressing and extracting files and directories.
The article introduces China’s AVS2 video codec, detailing its standards, technical implementation, code extensions for FLV and HLS, bitstream structure, performance comparisons with HEVC and x265, and JD Cloud’s current support and future plans for commercial deployment.
This article explains how enterprises can dramatically reduce cloud‑native data‑warehouse expenses by understanding ADB PG/Greenplum architecture, applying disk‑reservation and lock‑write safeguards, and implementing practical optimizations such as table compression, hot‑cold tiering, vacuuming, redundant‑index cleanup, replication conversion, and isolated temporary‑table spaces.
This article explains how Elasticsearch uses inverted indexes, term dictionaries, and compression techniques like FOR and Roaring Bitmaps to enable rapid full‑text search, contrasting its approach with traditional relational databases and offering practical indexing tips for large‑scale applications.
This article explains how Elasticsearch leverages Lucene’s inverted index, detailing the structure of term dictionaries, postings lists, compression techniques like Frame‑of‑Reference and Roaring Bitmaps, and query optimizations such as filter caches and skip‑list intersections to achieve fast, memory‑efficient search.
This article explains how Elasticsearch uses inverted indexes, term dictionaries, and compression techniques such as Frame‑of‑Reference and Roaring Bitmaps to deliver rapid full‑text search, efficient storage, and fast union queries, while also offering practical indexing tips for production use.
HEIF (High Efficiency Image File Format) offers superior compression, support for image sequences, depth and transparency, and is natively supported on iOS and Android P, with implementation options via ImageDecoder, BitmapFactory, or conversion to JPEG, though adoption faces compatibility and hardware licensing challenges.
The article analyzes the high cost and scalability challenges of traditional log collection pipelines and proposes a streamlined architecture that uses in‑memory buffering, UDP transport, aggressive compression, and ClickHouse storage to achieve massive throughput while drastically reducing hardware and operational expenses.
This article explains how Elasticsearch uses inverted indexes, term dictionaries, and advanced compression techniques like Frame of Reference and Roaring Bitmaps to enable rapid, scalable search over massive datasets, contrasting its approach with traditional relational database queries and detailing practical optimization tips.
This guide explains how to use OpenResty’s Lua modules to let Nginx read and write Redis directly, compress large payloads, schedule periodic cache updates, and configure URL‑based caching, thereby reducing latency, saving bandwidth, and improving overall backend reliability.
The article explains how a senior architect uses OpenResty and Lua to let Nginx directly access Redis for caching, compresses large responses, schedules periodic updates, and provides a configurable, signature‑verified caching layer to improve web performance and resilience.
This article explains how Elasticsearch uses inverted indexes, term dictionaries, and postings lists—along with compression techniques like Frame of Reference and Roaring Bitmaps—to achieve fast, memory‑efficient search queries, and provides practical tips for optimizing indexing and query performance.
This article explains how Elasticsearch uses inverted index structures—including term dictionaries, term indexes, and postings lists—combined with compression methods like Frame‑of‑Reference and Roaring Bitmaps to achieve fast search, efficient storage, and effective union queries compared to traditional relational databases.
This article explains how Elasticsearch uses inverted indexes, term dictionaries, postings lists, and compression techniques like Frame‑of‑Reference and Roaring Bitmaps to deliver rapid, memory‑efficient search, and provides practical tips for indexing and query optimization.
This article explains how Elasticsearch uses inverted indexes, term dictionaries, FST structures, and compression techniques like FOR and Roaring Bitmaps to dramatically speed up search queries over massive datasets while minimizing memory and disk usage.
This article explains how Elasticsearch achieves rapid search by using inverted indexes, detailing the structure of posting lists, term dictionaries, compression methods like Frame‑of‑Reference and Roaring Bitmaps, and how these techniques enable efficient union queries and filter caching.
This article explains how Elasticsearch leverages Lucene's inverted index, term dictionary, term index, and compression methods such as Frame‑of‑Reference and Roaring Bitmaps to enable rapid search, efficient storage, and fast set operations for large‑scale data retrieval.
This article explains how Elasticsearch achieves fast data retrieval by comparing it with traditional relational databases, detailing search engine fundamentals, the structure of Lucene's inverted index—including term dictionaries, postings lists, and term indexes—and the compression techniques such as Frame of Reference and Roaring Bitmaps that optimize storage and query performance.
The first Shenzhen Elasticsearch meetup on August 21, 2021, jointly hosted by the ES Chinese community and Tencent Cloud, gathered experts from Tencent, Tapdata, ByteDance and Vivo to showcase rapid community growth, compression‑encoding optimizations, real‑time ES‑MongoDB data fusion, custom kernel extensions, large‑scale cluster practices, and concluded with extensive Q&A and networking.
This article explains the glTF format, compares .gltf and .glb, shows how to split and compress glTF assets using tools like gltf‑pipeline, Draco, mesh quantization and meshopt, and presents performance tests across various devices to guide efficient 3D model delivery in web applications.
This article explains how to use OpenResty with Lua to build a high‑performance caching layer that directly accesses Redis, compresses large responses, schedules periodic updates, forwards requests intelligently, and provides configurable URL caching, thereby improving concurrency, reducing latency, and optimizing bandwidth usage.
This article provides a comprehensive introduction to Apache Kafka, covering its core concepts, terminology, data flow, configuration options such as compression and partitioning, producer and consumer behavior, replication, consumer groups, offset management, and log retention strategies.
This article explains how Elasticsearch leverages inverted indexes, term dictionaries, and advanced compression techniques like Frame of Reference and Roaring Bitmaps to enable rapid full‑text search, covering the underlying concepts, data structures, and query optimizations essential for high‑performance search applications.
This article explains how Elasticsearch achieves fast search by using inverted indexes, detailing the structure of posting lists, term dictionaries, term indexes, and compression methods such as Frame of Reference and Roaring Bitmaps, as well as techniques for efficient union and intersection queries.
The article analyzes WeChat Moments' massive image and video services, detailing its OC/IDC architecture, holiday traffic challenges, software and hardware safeguards, disaster‑recovery mechanisms, retry policies, and a series of flexible strategies—including compression format changes, bitrate reduction, buffer pools, and timeline throttling—to sustain billions of daily accesses.
This article explains why MySQL binlog files can become large, demonstrates how to enable the slave_compressed_protocol parameter for binlog compression, presents benchmark results showing reduced network traffic and CPU impact, and outlines compatibility considerations and known bugs.
This article explains how digital images are captured as pixel arrays, describes the physiological basis of color perception, compares RGB and YUV color spaces, and introduces video compression concepts such as spatial and temporal redundancy, frame types, and common encoding formats.
The PHP function zip_entry_compressionmethod() returns the compression method (such as STORE or DEFLATE) used for a specific zip entry resource obtained via zip_read(), with a clear description of its parameter and return value.
The iQIYI International backend team boosted global video delivery by introducing a web‑cache layer, Brotli compression, and a two‑level Caffeine‑Redis caching system with pub/sub synchronization, achieving up to 6% faster First Contentful Paint, near‑100% cache‑hit rates, and measurable revenue gains.
This guide provides a comprehensive overview of common Linux compression and decompression utilities—including gzip, bzip2, zip/unzip, xz, and tar—detailing their descriptions, key options, usage examples, and tips for handling files and directories efficiently.
This article explains how to enable and fine‑tune Nginx’s Gzip compression, detailing the benefits for CSS, JS, XML and HTML delivery, providing configuration directives, recommended parameter values, and example snippets for both local and production environments.
Learn how to use the Linux tar command to archive files and directories, explore options for creating, listing, and extracting archives, and compare gzip, bzip2, and xz compression methods with performance metrics and practical examples.
By integrating SIMD‑accelerated ISA‑L and parallel Zstandard (Pzstd) into Meituan’s build‑and‑deployment pipeline, compression time for typical 200 MiB–1 GiB packages dropped from over a minute to about one second—a 90%‑plus speedup that preserves gzip compatibility while dramatically accelerating overall build latency.
This guide explains how to install and use the zip, unzip, and tar commands on Linux to compress files, create archives, list contents, extract data, and apply gzip, bzip2, or xz compression with various options and examples.
The article explains why Hive tables generate many small files on HDFS, describes the performance impact on NameNode and MapReduce, and provides detailed configuration steps and compression techniques—including input and output file merging, various Hive file formats, and partition optimization—to efficiently manage storage and resource consumption in big‑data environments.
The article evaluates how turning on binlog transaction compression in MySQL 8.0 (both 8.0.19 and 8.0.20) affects throughput and resource usage under a sysbench workload of 120 tables with 240 concurrent threads, showing roughly a 1% performance drop and a similar increase in CPU consumption.
This article introduces JD Cloud’s self‑built time‑series database HoraeDB, explaining its core concepts, typical use cases, architectural layers, high‑performance features, down‑sampling strategies, compression techniques, and stability measures for handling massive, 24‑hour monitoring data at scale.
This guide explains how to enable LZO compression in Hadoop by installing the twitter‑provided hadoop‑lzo library, updating core‑site.xml, synchronizing files across nodes, creating LZO indexes, and running a WordCount MapReduce job with LZO‑compressed output.
The article explains SAP HANA’s performance advantages by combining four key technologies—high‑speed memory, columnar storage, data compression, and an insert‑only model—detailing their individual pros and cons, how they complement each other, and the trade‑offs involved in scaling and persistence.
Web performance is critical in today's online economy; this article explains ten practical NGINX‑based techniques—including reverse proxy, load balancing, caching, compression, SSL/TLS tuning, HTTP/2, software upgrades, Linux tweaks, server tuning, and real‑time monitoring—that together can deliver up to tenfold speed improvements and higher conversion rates.
This article explains why web performance is critical, presents ten practical NGINX‑based recommendations—including reverse proxy, load balancing, caching, compression, SSL/TLS optimization, HTTP/2, software upgrades, Linux tuning, and real‑time monitoring—to help developers and operators dramatically improve application speed and reliability.
This article demonstrates how to use Java's Deflater and DeflaterOutputStream classes to compress data into Base64 strings, how to decompress them with InflaterOutputStream, and includes Spock test cases that compare compressed and original lengths for various inputs.
This article presents ten actionable recommendations—including reverse‑proxy deployment, load balancing, caching, compression, SSL/TLS tuning, HTTP/2 adoption, software upgrades, Linux and web‑server tuning, and real‑time monitoring—to dramatically improve web application performance, often achieving tenfold speed gains.
This article presents ten practical NGINX‑based techniques—including reverse proxy deployment, load balancing, caching, compression, SSL/TLS optimization, HTTP/2 adoption, software upgrades, Linux tuning, web‑server configuration, and real‑time monitoring—that together can boost web‑application throughput and latency by up to an order of magnitude.
This article introduces TimescaleDB—a PostgreSQL extension for time‑series data—explaining its core concepts, data‑model choices, architecture, native compression, and provides step‑by‑step installation instructions on a CentOS 7 environment.
This article explains why web performance matters, presents ten practical NGINX‑based techniques—including reverse proxy, load balancing, caching, compression, SSL/TLS optimization, HTTP/2, software upgrades, Linux and web‑server tuning, and real‑time monitoring—to help you achieve up to tenfold speed improvements.
The article explains MySQL 8.0.20's binlog transaction compression feature using the ZSTD algorithm, describes its internal implementation, outlines usage steps, presents performance test results, and lists important considerations for deploying compression in production environments.
This article describes ByteDance’s extensive improvements to the RocksDB storage engine—including LazyBuffer, Adaptive Map‑based lazy compaction, KV separation, adaptive multi‑index support, extreme compression techniques, and hardware acceleration—to reduce amplification, improve performance, and lower costs for large‑scale database workloads.
Explore the essential building blocks of modern computing, covering CPU architecture, memory hierarchy, binary operations, compression techniques, operating system fundamentals, and assembly language, with clear explanations, diagrams, and code examples that demystify how hardware and software interact at the lowest level.
This article explains the ORC (Optimized Record Columnar) file format used in Hive, covering its architecture, stripe and column storage, handling of complex data types, indexing mechanisms, compression streams, memory management, and key configuration parameters.
The article outlines four key dimensions for extreme HTTP performance optimization—encoding efficiency, channel utilization, transmission path, and information security—explaining how advances like binary encoding, multiplexed streams, TCP/QUIC tuning, and TLS 1.3 together reduce latency, boost concurrency, and enhance user experience.
This article demonstrates how to dramatically speed up Java file compression of multiple images by replacing naive FileInputStream reads with BufferedInputStream, then leveraging NIO Channels with transferTo, memory‑mapped files, and finally Pipe‑based streaming, showing performance improvements from 30 seconds down to about 1 second.
This article explains the PNG image format, covering its lossless compression, different PNG types (8, 24, 32), internal data structure, how prediction and Deflate compression work, and tips for achieving better compression by using simple, low‑color images.
This guide presents ten practical Linux commands—including Vim, tar, rar, unzip, zipinfo, zcat, zless, and less—that let you view the contents and metadata of archive or compressed files directly without extracting them first, saving time and effort.
Gaode’s map platform serializes massive geographic data by weighing bandwidth, extensibility, and decoding speed, comparing open‑source protobuf and FlatBuffers with a bespoke binary format that adds chapter‑based storage, variable attributes, and aggressive compression, ultimately selecting the method that best fits specific size‑or‑speed priorities.
This article examines how network bandwidth, protocol compression, and SSL encryption impact MySQL Group Replication performance by running sysbench benchmarks on 1 Gb and 10 Gb networks, revealing bandwidth limits, CPU overhead, and scalability issues.
This article shares practical strategies used in a multi‑petabyte Hadoop environment to slash storage expenses, covering reduced replication, selective compression formats, tiered storage policies, and erasure coding, while weighing trade‑offs in reliability, performance, and operational complexity.
This weekly bulletin summarizes four Hadoop knowledge points—compression formats, MapReduce join techniques, Hive installation, and YARN Capacity Scheduler—while also sharing personal updates about a PhD graduation, the upcoming May Day holiday, and a request for likes and shares.
Arch Linux maintainers replaced the single‑threaded xz compression in devtools with Facebook's fast, multithreaded Zstandard algorithm at level ‑18, achieving dramatically higher compression and decompression speeds and more stable parallel processing, which will speed up pacman package installations.
This article reveals several lesser‑known Python features—including functions that accept arbitrary arguments, powerful file searching with glob, debugging with inspect, generating unique IDs via uuid, data serialization, string compression, and registering shutdown hooks—providing practical code examples for each.
This article reviews major in‑memory inverted index compression techniques such as PForDelta, PEF, and MILC, explains their principles and trade‑offs, and details practical optimizations applied at 58.com to achieve query performance comparable to uncompressed indexes while reducing memory usage by about 35 percent.
The article recounts how a 1960 Playboy portrait of Lena Söderberg was adopted by image‑processing researchers as a standard test image, explains the technical and cultural reasons for its lasting popularity, and follows her unexpected rise to fame within the scientific community.
This article explains the concept of Node.js streams, why they improve I/O performance and memory usage, and provides step‑by‑step examples of readable, writable, and transform streams—including piping, back‑pressure handling, and gzip compression.
This article explains the core principles of RPC message protocol design, covering how to determine message boundaries, choose between delimiter and length‑prefix methods, handle explicit and implicit message structures, apply compression, and implement varint and zigzag encoding for optimal traffic efficiency.
This article details Lianjia's journey from a Hadoop‑based 0.0 data platform to a sophisticated 2.0 architecture, describing the three‑layer design, OLAP engine choices, transparent compression techniques, operational challenges, and practical recommendations for building and maintaining large‑scale big data systems.
This article explains the evolution of Kafka message formats from version 0.7.x through 0.8.x (including 0.9.x) to 0.10.x, detailing each field, compression handling, and the design motivations behind the changes.
The April 2018 Ceph monthly report highlights LTTng tracing for RGW, SSL support for the beast frontend, MFA integration, a notrim option for rbd mapping, runtime lz4 and brotli compression, Zabbix PG metrics, asynchronous dashboard tasks, detailed operation tracking, osdmap pruning, and a new iostat manager plugin.
This article explains the history, data layout, encoding, compression techniques, and real‑world implementations of column‑oriented storage, showing how it reduces I/O and improves cache efficiency for analytical workloads while highlighting its trade‑offs for transactional use.
This article explains the fundamentals of web image optimization, covering vector vs raster formats, compression methods, responsive image techniques with srcset and picture elements, and practical loading strategies such as lazy loading, sprites, and progressive JPEGs to improve front‑end performance.
This guide explains how gzip and deflate compression work in web browsers and demonstrates straightforward Node.js examples for compressing and decompressing files, streaming responses, and handling HTTP requests with proper Accept‑Encoding checks, helping developers improve bandwidth efficiency and page load speed.
This article explores practical lossy and lossless compression techniques for web front‑end assets, covering image compression via Canvas, data compression using LZ‑based algorithms, recommended JavaScript libraries, and a novel PNG‑based approach to shrink resources without sacrificing functionality.
This article explains how to detect the EXIF Orientation tag on mobile‑captured photos, rotate them correctly with canvas, and compress the result to reduce size, providing a lightweight solution for fixing 90° rotation issues on iPhone and certain Android devices.
This article explains the technical distinctions between data deduplication and compression, compares their use in backup versus primary storage environments, and details how major vendors implement these technologies in SSD and flash arrays, highlighting performance, architectural, and operational considerations.
This article examines how texture dimensions, data formats, and compression techniques influence creation time, memory usage, and rendering performance in WebGL, offering practical guidelines for optimizing texture handling in web graphics applications.
This article explores how the ordering of HTML and CSS attributes influences gzip compression ratios, demonstrates size differences with reordered code, and clarifies Chrome DevTools' Size vs. Content values, offering practical tips for better web performance and maintainable code.
This article provides a comprehensive overview of RocksDB, covering its origin, design goals, core architecture components, key features such as APIs, compression strategies, durability mechanisms, backup and replication support, as well as tooling, testing, and performance characteristics.
The benchmark shows that while InnoDB delivers roughly 30 % higher insert throughput than TokuDB, the TokuDB engine achieves 10‑16× storage compression with comparable import times, minimal impact from log‑sync settings, and zlib emerging as the preferred compression algorithm for balanced performance and CPU usage.
This guide provides concise descriptions, syntax, key parameters, and usage examples for a range of essential Linux commands—including ar, bunzip2, bzip2, gunzip, tar, zip, and others—helping users efficiently manage archives, compression, and system backups.
This article provides a comprehensive guide to common Linux commands, covering internal vs external classification, help lookup methods, command syntax, practical examples for file management, system monitoring, compression, shutdown, pipelines, quoting, and best learning habits.
PostgreSQL's TOAST mechanism automatically compresses and, when necessary, moves oversized column values to an external toast table, using four storage strategies (plain, extended, external, main) that dictate compression and externalization behavior, with data exceeding roughly 2 KB triggering external storage regardless of strategy.
This article explains Oracle 12c's In‑Memory Column Store feature, its compression levels, step‑by‑step testing results for full scans and joins, a real‑world production case from Zhejiang Mobile, and practical maintenance commands to keep the in‑memory area healthy.
