Understanding Spark Memory Management: On‑heap, Off‑heap, and Unified Memory

This article provides a comprehensive overview of Spark's memory management, covering executor memory architecture, the differences between on‑heap and off‑heap memory, static versus unified memory managers, storage and execution memory handling, and practical guidelines for optimizing Spark applications.

Big DataExecutorMemory Management

0 likes · 21 min read

Understanding Spark Memory Management: On‑heap, Off‑heap, and Unified Memory

Big Data Technology Architecture

Apr 16, 2020 · Databases

Memory Management Optimizations in HBase MemStore: SkipList, MemStoreLAB, ChunkPool, Off‑heap and CCSMap

The article systematically explains how HBase's MemStore uses a SkipList‑based model and introduces successive memory‑management optimizations—MemStoreLAB, ChunkPool, off‑heap chunks, CompactingMemStore and the CCSMap data structure—to reduce object overhead, GC pressure and improve throughput.

HBaseJavaMemStore

0 likes · 20 min read

Memory Management Optimizations in HBase MemStore: SkipList, MemStoreLAB, ChunkPool, Off‑heap and CCSMap