Memory Optimization Deep Dive: Allocate Less, Allocate Faster, Use Efficiently

Three Directions of Memory Optimization

Memory optimization fundamentally addresses three goals: allocate less , allocate fast , and use efficiently . These directions are interconnected—reducing allocations often improves cache hit rates, while faster allocation reduces fragmentation and stabilizes latency.

Allocate Less: Reduce Allocation Frequency at the Source

(1) Prefer Stack Allocation, Avoid Heap

Stack allocation is nearly zero-cost (just moving the stack pointer) and automatically reclaimed on function exit with no fragmentation. Small objects and temporaries should use stack.

// Bad: frequent heap allocation for small object
std::string* s = new std::string("hello");
delete s;

// Good: direct stack allocation
std::string s = "hello";

(2) Pre-allocate Containers to Avoid Repeated Reallocations

std::vector

and std::string reallocate by requesting larger memory, copying old data, and freeing old memory. reserve() turns multiple reallocations into one.

std::vector<int> v;
v.reserve(10000);  // Pre-allocate space, subsequent push_back won't reallocate
for (int i = 0; i < 10000; ++i) {
    v.push_back(i);
}

(3) Reuse Memory with Object Pools / Memory Pools

High-frequency small objects (network packets, task nodes) should not use new/delete each time. Pre-allocate a large block and manage a free list; allocation and deallocation become O(1).

class Packet {
public:
    static void* operator new(size_t) {
        if (!free_list) {
            // Allocate a large chunk once, slice into N Packet-sized blocks
            free_list = allocate_chunk();
        }
        void* p = free_list;
        free_list = free_list->next;
        return p;
    }
    static void operator delete(void* p) {
        // Return to free list instead of system
        ((Node*)p)->next = free_list;
        free_list = (Node*)p;
    }
private:
    struct Node { Node* next; };
    static Node* free_list = nullptr;
};

Why it works: malloc/free requires locking, searching free blocks, and merging fragments—far more overhead than popping a node from a free list.

(4) Release Large Objects Immediately, Don't Wait for GC / Destructor

C++ has no GC; release timing is fully under your control. Large arrays/buffers should be released immediately after use via clear() + shrink_to_fit() or by placing them in an inner scope for automatic destruction.

#include <iostream>
#include <vector>
#include <string>

int main() {
    // Method 1: inner scope, auto-destruct on exit
    {
        std::vector<char> big_buf(1024 * 1024 * 128); // 128MB buffer
        // use big_buf ...
    } // vector destructs, heap memory released

    // Method 2: explicit release without scope
    std::vector<char> buf(1024 * 1024 * 128);
    // use buf ...
    buf.clear();
    buf.shrink_to_fit(); // clear only removes elements; shrink_to_fit returns heap memory to allocator

    std::string s(1024 * 1024 * 64, 'x');
    // use s ...
    s.clear();
    s.shrink_to_fit();
    return 0;
}

Key point: clear() only sets size to 0; capacity unchanged, heap memory not returned. Must pair with shrink_to_fit() to release underlying memory.

(5) Use Move Semantics to Avoid Deep Copies

When passing std::vector or std::string between functions, use std::move to transfer heap memory ownership instead of copying all data.

#include <iostream>
#include <vector>
#include <string>
#include <utility> // std::move
// Pass by value: triggers copy (deep copy)
void pass_copy(std::vector<int> v) {
    v.push_back(999);
}
// Rvalue reference: takes ownership, only transfers metadata, no data copy
void pass_move(std::vector<int>&& v) {
    v.push_back(999);
}
int main() {
    std::vector<int> vec{1,2,3,4,5};
    std::string s("hello world, long string...");
    // Deep copy: allocates new heap memory, copies all elements
    pass_copy(vec);
    std::cout << "after copy, vec still valid, size: " << vec.size() << "
";
    // std::move casts to rvalue, move semantics: only transfers internal pointer, size, capacity; heap data not copied
    pass_move(std::move(vec));
    std::cout << "after move, vec in valid but unspecified state, size: " << vec.size() << "
";
    std::string s2 = std::move(s);
    std::cout << "s2: " << s2 << ", original s is empty: " << s << "
";
    return 0;
}

Move semantics transfers internal heap memory ownership via std::move, moving only metadata (pointer, length, capacity) and avoiding expensive deep copies. The moved-from object is left in a valid but unspecified state; do not rely on its original data.

Tip: Returning local containers triggers compiler RVO / move automatically; writing return std::move(xxx) manually can inhibit RVO.

Allocate Fast: Make Memory Layout Friendly

(1) Classify by Object Size, Prevent Small Objects from Fragmenting the Heap

Massive numbers of 16-byte, 32-byte objects repeatedly allocated/freed are the primary source of heap fragmentation. Solution: manage by size class—16-byte objects go to one pool, 64-byte to another, no interference.

#include <stdio.h>
#include <stdlib.h>
#include <stdint.h>
#include <string.h>

// Size class list, powers of two like tcmalloc
#define SIZE_CLASS_CNT 4
static const size_t size_classes[SIZE_CLASS_CNT] = {16, 32, 64, 128};

// Free block header; block itself stores next pointer, memory reused
typedef struct free_node {
    struct free_node *next;
} free_node_t;

// Each bucket: free list head
static free_node_t *bucket[SIZE_CLASS_CNT] = {NULL};

// Find matching size class index, round up
static int get_size_class_idx(size_t sz)
{
    for (int i = 0; i < SIZE_CLASS_CNT; i++) {
        if (sz <= size_classes[i]) {
            return i;
        }
    }
    return -1; // Beyond pool range, fall back to system malloc
}

// Allocate from corresponding bucket
void *small_alloc(size_t sz)
{
    int idx = get_size_class_idx(sz);
    if (idx < 0) {
        return malloc(sz); // Large object goes to system
    }

    free_node_t *p = bucket[idx];
    if (p != NULL) {
        bucket[idx] = p->next;
        return p;
    }

    // Free list empty: batch allocate from system (reduce syscalls)
    size_t chunk_size = size_classes[idx];
    const int batch = 8;
    uint8_t *base = malloc(batch * chunk_size);
    if (base == NULL) return NULL;

    // Link new memory into free list
    for (int i = 1; i < batch; i++) {
        free_node_t *node = (free_node_t *)(base + i * chunk_size);
        node->next = bucket[idx];
        bucket[idx] = node;
    }
    return base;
}

// Return memory to corresponding bucket
void small_free(void *ptr, size_t sz)
{
    if (ptr == NULL) return;
    int idx = get_size_class_idx(sz);
    if (idx < 0) {
        free(ptr);
        return;
    }
    free_node_t *node = (free_node_t *)ptr;
    node->next = bucket[idx];
    bucket[idx] = node;
}

// Test
int main(void)
{
    void *p1 = small_alloc(16);
    void *p2 = small_alloc(32);
    void *p3 = small_alloc(10); // Rounds up to 16-byte bucket
    void *p4 = small_alloc(50); // Rounds up to 64-byte bucket

    printf("p1=%p, p2=%p, p3=%p, p4=%p
", p1, p2, p3, p4);

    small_free(p1, 16);
    small_free(p2, 32);
    small_free(p3, 10);
    small_free(p4, 50);

    // Re-allocate: reuses just-freed memory, no malloc call
    void *p1x = small_alloc(16);
    printf("reuse p1x=%p
", p1x);
    return 0;
}

Each size class maintains an independent free list; 16B blocks never mix into 32B bucket, eliminating external fragmentation from small-block splitting. Free blocks reuse their own space for the next pointer—zero overhead. When a list is exhausted, batch allocation from system reduces malloc calls. Objects exceeding max size-class go directly to system malloc.

Drawbacks: not thread-safe; multi-threaded use requires thread-local caches with locks or lock-free queues (origin of tcmalloc thread cache). Internal fragmentation occurs—e.g., requesting 10B consumes a 16B block—trading internal fragmentation to avoid external heap fragmentation.

(2) Large Contiguous Allocation to Reduce Page Table Overhead

A single mmap of 1GB contiguous memory uses fewer page table entries and achieves higher TLB hit rates than 1000 separate 1MB mmap calls. Linux HugePages (2MB or 1GB pages) further reduce TLB misses.

#include <stdio.h>
#include <stdlib.h>
#include <unistd.h>
#include <sys/mman.h>
#include <errno.h>

// Example: single large mmap
#define ONE_BLOCK_SIZE (1LU << 30)  // 1GB
#define SMALL_BLOCK_SIZE (1LU << 20) // 1MB
#define SMALL_CNT 1000

int main(void) {
    // Option 1: allocate 1GB contiguous
    void *big_buf = mmap(NULL, ONE_BLOCK_SIZE,
                         PROT_READ | PROT_WRITE,
                         MAP_PRIVATE | MAP_ANONYMOUS,
                         -1, 0);
    if (big_buf == MAP_FAILED) {
        perror("mmap big");
        return 1;
    }
    printf("big buf addr: %p, size: 1G
", big_buf);

    // Use: manually slice this contiguous space into smaller blocks
    char *p = (char*)big_buf;
    for (int i = 0; i < SMALL_CNT; ++i) {
        *(p + i * SMALL_BLOCK_SIZE) = 1;
    }

    // Option 2 (comparison): scattered 1000 x 1MB mmap
    // Consumes more page table entries, higher TLB pressure
    void *small_ptrs[SMALL_CNT];
    for (int i = 0; i < SMALL_CNT; ++i) {
        small_ptrs[i] = mmap(NULL, SMALL_BLOCK_SIZE,
                             PROT_READ | PROT_WRITE,
                             MAP_PRIVATE | MAP_ANONYMOUS,
                             -1, 0);
        if (small_ptrs[i] == MAP_FAILED) {
            perror("mmap small");
            break;
        }
    }

    // Cleanup
    munmap(big_buf, ONE_BLOCK_SIZE);
    for (int i = 0; i < SMALL_CNT; ++i) {
        if (small_ptrs[i]) munmap(small_ptrs[i], SMALL_BLOCK_SIZE);
    }
    return 0;
}

HugePage version (requires system pre-configuration):

// 2MB hugepage, add MAP_HUGETLB flag
#define HUGEPAGE_2M (2LU << 20)
void *huge_buf = mmap(NULL, HUGEPAGE_2M,
                      PROT_READ | PROT_WRITE,
                      MAP_PRIVATE | MAP_ANONYMOUS | MAP_HUGETLB,
                      -1, 0);

Note: HugePages must be reserved in system config beforehand; cannot be used directly. Transparent Huge Pages (THP) require no code changes but offer weaker controllability than static HugePages.

Use Efficiently: Make Every Byte Count

(1) Choose Appropriate Data Types

Developers often default to int for everything, but most fields don't need full 32-bit range. A status field with only a few enum values can use uint8_t (1 byte) instead of int (4 bytes). Ten such fields save 30 bytes per object; with millions of objects, that's tens of MB.

// Bad: all fields use int
struct OrderBad {
    int order_id;      // 4 bytes, but only up to 10 million
    int status;        // 4 bytes, only 5 states
    int user_type;     // 4 bytes, only 3 types
    int priority;      // 4 bytes, range 0-9
};

// Good: choose type by actual range
struct OrderGood {
    uint32_t order_id; // 4 bytes, holds 4 billion
    uint8_t  status;   // 1 byte, holds 255 states
    uint8_t  user_type; // 1 byte
    uint8_t  priority;  // 1 byte
    uint8_t  _padding;  // 1 byte for alignment
};

Size drops from 16 to 8 bytes. Smaller structs fit more objects per cache line, speeding up traversal.

(2) Mind Struct Alignment

Compilers insert padding bytes between members for access efficiency. Poor member ordering wastes space.

// Bad order: holes in the middle
struct Bad {
    char  a;   // 1 byte
    int   b;   // 4 bytes, 3 bytes padding before
    char  c;   // 1 byte, 3 bytes padding after
    short d;   // 2 bytes
};
// sizeof(Bad) = 12 bytes, only 8 bytes useful

// Good order: largest to smallest
struct Good {
    int   b;   // 4 bytes
    short d;   // 2 bytes
    char  a;   // 1 byte
    char  c;   // 1 byte
};
// sizeof(Good) = 8 bytes, perfectly packed

Simple rule: Put larger members first, smaller later. Compiler inserts less padding.

(3) Align Memory to Avoid False Sharing

In multi-core scenarios, two frequently modified variables on different threads sharing a cache line (typically 64 bytes) cause cache line invalidation ping-pong—false sharing. Use alignas to separate them.

#include <iostream>
#include <thread>
#include <atomic>
// Typical CPU cache line 64 bytes
#define CACHE_LINE_SIZE 64
// False sharing: two atomics in same cache line
struct Bad {
    std::atomic<uint64_t> a;
    std::atomic<uint64_t> b;
};
// Fixed: alignas forces each variable to own a cache line
struct Good {
    alignas(CACHE_LINE_SIZE) std::atomic<uint64_t> a;
    alignas(CACHE_LINE_SIZE) std::atomic<uint64_t> b;
};

template<typename T>
void bench() {
    T data{};
    auto t1 = std::thread([&](){
        for(int i = 0; i < 100000000; ++i) data.a.fetch_add(1);
    });
    auto t2 = std::thread([&](){
        for(int i = 0; i < 100000000; ++i) data.b.fetch_add(1);
    });
    t1.join();
    t2.join();
    std::cout << "a=" << data.a << ", b=" << data.b << '
';
}
int main() {
    std::cout << "Bad (false sharing):
";
    bench<Bad>();
    std::cout << "Good (fixed with alignas):
";
    bench<Good>();
    return 0;
}

Frequent modifications by different cores to different variables in the same cache line trigger repeated invalidations—false sharing. alignas(64) aligns variable start addresses to 64 bytes, giving each its own cache line and preventing mutual invalidation.

Addition: C++17 provides std::hardware_destructive_interference_size as a portable alternative to hard-coded 64:

#include <new>
alignas(std::hardware_destructive_interference_size) std::atomic<uint64_t> a;

In summary, allocate less controls frequency, allocate fast stabilizes latency, use efficiently improves throughput . Memory optimization is not single-parameter tuning but end-to-end engineering governance across allocation, allocation speed, and access. Combine strategies based on business characteristics to achieve lower latency, higher throughput, and more stable long-term operation within limited memory resources.