Understanding Data Deduplication: Definitions, Classifications, and Its Relationship with Compression

Deduplication and compression are popular techniques for efficiently saving storage space and are widely used in main memory, backup software, and flash storage systems; the following discusses the concept of deduplication and its various classifications.

1. Definition of Data Deduplication According to Wikipedia, data deduplication is a technology that saves storage space by storing only one copy of duplicate data. Unlike data compression, which searches for duplicate data at a fine granularity of a few bits to bytes, deduplication operates on larger blocks (typically 1 KB or larger) to identify and eliminate redundancy.

2. Classification of Data Deduplication

By execution time: Inline deduplication processes data before it is written to disk, while Postline deduplication writes data first and then reads it back for deduplication, typically scheduled during low‑load periods.

By block size: Fixed‑length deduplication divides data into equal‑sized chunks before deduplication, whereas Variable‑length deduplication creates chunks of different sizes, often yielding better deduplication ratios in backup scenarios.

By granularity: Block‑level deduplication computes fingerprints for individual data blocks, while File‑level deduplication (also called single‑instance storage) computes a fingerprint for an entire file. Both approaches can be combined within a unified storage system.

By location: Source‑side deduplication performs chunking and fingerprinting at the source and only transmits unique blocks to the target, reducing network bandwidth. Target‑side deduplication receives raw data first, then performs chunking and fingerprinting on the target side. In practice, many deployments use a mix of these methods.

3. Difference and Relationship Between Deduplication and Compression Compression works at the byte level using encoding schemes such as Huffman coding, while deduplication relies on hash algorithms to identify duplicate blocks. Deduplication can be viewed as block‑level compression, and the two techniques are often used together—for example, after deduplication, unique blocks may be further compressed to achieve maximal storage reduction.

Warm tip: Scan the QR code below to follow the public account for more content.