Boost C++ Service Performance: 3× Faster with Classes, Cache‑Friendly Structures, jemalloc and Lock‑Free Designs

Performance‑driven refactoring of a high‑throughput C++ service

Large‑scale C++ services often suffer from hidden costs caused by heavyweight abstractions, sub‑optimal data layouts and the default memory allocator. By profiling hot paths and measuring concrete metrics, a series of targeted changes reduced latency and memory consumption while keeping the code maintainable.

1. Replace Protobuf with a hand‑written class

Protobuf messages are allocated from an arena allocator, which creates many small heap allocations and incurs costly destructor work, especially for string fields. The original definition:

message Param {
    optional string name = 1;
    optional string value = 2;
}

message ParamHit {
    optional Param param = 1;
    optional uint64 group_id = 2;
    // … other fields …
}

was rewritten as a plain C++ class that owns its std::string members and provides explicit Clear methods:

class ParamHitInfo {
public:
    ParamHitInfo() = default;
    const std::string& name() const { return name_; }
    void set_name(const std::string& n) { name_ = n; }
    // similar getters/setters for value, ids, etc.
private:
    std::string name_, value_;
    uint64_t group_id_, expt_id_, launch_layer_id_;
    bool is_hit_mbox_;
    Param param_;
};

A micro‑benchmark created 1,000 ParamHit objects, copied them 1,000 times and measured destructor time. The class‑based version was ≈3× faster than the Protobuf version, confirming that eliminating arena allocations removes a major overhead.

2. Use cache‑friendly containers

Although hash tables have O(1) lookup complexity, their random memory accesses hurt CPU cache locality. The original HitContext stored all key‑value pairs in an std::unordered_map:

class HitContext {
public:
    void update_hash_key(const std::string& key, const std::string& val) {
        hash_keys_[key] = val;
    }
    const std::string* search_hash_key(const std::string& key) const {
        auto it = hash_keys_.find(key);
        return it != hash_keys_.end() ? &it->second : nullptr;
    }
private:
    std::unordered_map<std::string, std::string> hash_keys_;
};

For a workload where a large fraction of keys follow the pattern "sns"+id, a hybrid design was introduced. Special keys are stored in a dense std::vector<std::pair<uint32_t,uint32_t>> while all other keys remain in the unordered map:

class HitContext {
public:
    void update_hash_key(const std::string& key, const std::string& val) {
        if (Misc::IsSnsHashKey(key)) {
            uint32_t id = Misc::FastAtoi(key.c_str() + Misc::SnsHashKeyPrefix().size());
            sns_hash_keys_.emplace_back(id, Misc::LittleEndianBytesToUInt32(val));
            return;
        }
        hash_keys_[key] = val;
    }
    std::string search_hash_key(const std::string& key, bool& find) const {
        if (Misc::IsSnsHashKey(key)) {
            auto it = std::find_if(sns_hash_keys_.rbegin(), sns_hash_keys_.rend(),
                                   [key](const auto& v){ return v.first == Misc::FastAtoi(key.c_str()+Misc::SnsHashKeyPrefix().size()); });
            find = it != sns_hash_keys_.rend();
            return find ? Misc::UInt32ToLittleEndianBytes(it->second) : "";
        }
        auto it = hash_keys_.find(key);
        find = it != hash_keys_.end();
        return find ? it->second : "";
    }
private:
    std::unordered_map<std::string, std::string> hash_keys_;
    std::vector<std::pair<uint32_t, uint32_t>> sns_hash_keys_;
};

Micro‑benchmarks (10 µs‑level) showed a ≈30 % reduction in lookup time compared with the pure unordered‑map version, demonstrating the benefit of tailoring the container to the key distribution.

3. Switch to a high‑performance allocator (jemalloc/tcmalloc)

The default new/delete implementation in many STL containers suffers from fragmentation, poor cache friendliness and a global lock under contention. Adding a dependency on //mm3rd/jemalloc:jemalloc in a Bazel target links the service against jemalloc:

cc_library(
    name = "mmexpt_dye_api",
    srcs = ["mmexpt_dye_api.cc"],
    hdrs = ["mmexpt_dye_api.h"],
    deps = ["//mm3rd/jemalloc:jemalloc"],
    copts = ["-O3", "-std=c++11"],
    visibility = ["//visibility:public"],
)

Real‑world load‑business benchmarks (request‑path latency) recorded a ≈20 % latency reduction after enabling jemalloc, confirming the theoretical advantages of a thread‑caching allocator.

4. Lock‑free double‑buffer for extreme write‑read concurrency

When the API processes up to 2.6 billion calls per second, a lock‑free design is required. The solution uses two memory buffers, a volatile switch flag, a CRC checksum and a multi‑level hash table:

struct expt_api_new_shm {
    void* p_shm_data;
    volatile int* p_mem_switch; // 0: uninit, 1: mem1 active, 2: mem2 active
    uint32_t* p_crc_sum;
    expt_new_context* p_new_context;
    parameter2business* p_param2business;
    char* p_business_cache;
    HashTableWithCache hash_table;
};

int InitExptNewShmData(expt_api_new_shm* shm, void* data) { /* map layout, init hash table */ }
void SwitchNewShmMemToWrite(expt_api_new_shm* shm) { /* point to inactive buffer */ }
void SwitchNewShmMemToWriteDone(expt_api_new_shm* shm) { /* flip switch flag */ }
void SwitchNewShmMemToRead(expt_api_new_shm* shm) { /* point readers to the freshly written buffer */ }

Because readers and writers operate on different buffers, no mutex is required. The trade‑off is a 2× memory footprint, which is acceptable for the latency‑critical path.

5. Tailor data formats to the use case

The original expt_param_item struct contained many fields that the downstream "dye" scenario never used, inflating both serialization time and memory usage:

struct expt_param_item {
    int experiment_id;
    int expt_group_id;
    int layer_id;
    int domain_id;
    uint32_t seq;
    uint32_t start_time;
    uint32_t end_time;
    uint8_t expt_type;
    uint16_t expt_client_expand;
    int parameter_id;
    uint8_t value[MAX_PARAMETER_VLEN];
    char param_name[MAX_PARAMETER_NLEN];
    int value_len;
    uint8_t is_pkg;
    uint8_t is_white_list;
    uint8_t is_launch;
    uint64_t bucket_src;
    uint8_t is_control;
};

Only experiment_id, expt_group_id and bucket_src are required. A minimal struct was introduced:

struct DyeHitInfo {
    int expt_id, group_id;
    uint64_t bucket_src;
    DyeHitInfo() {}
    DyeHitInfo(int e, int g, uint64_t b) : expt_id(e), group_id(g), bucket_src(b) {}
    bool operator<(const DyeHitInfo& o) const {
        if (expt_id != o.expt_id) return expt_id < o.expt_id;
        if (group_id != o.group_id) return group_id < o.group_id;
        return bucket_src < o.bucket_src;
    }
    bool operator==(const DyeHitInfo& o) const {
        return expt_id == o.expt_id && group_id == o.group_id && bucket_src == o.bucket_src;
    }
    std::string ToString() const {
        char buf[1024];
        sprintf(buf, "expt_id: %u, group_id: %u, bucket_src: %lu", expt_id, group_id, bucket_src);
        return std::string(buf);
    }
};

Benchmarks on the production pipeline showed a 40 % reduction in serialization time and a **30 %** drop in memory consumption compared with the full Protobuf payload.

6. Systematic performance testing

To reliably detect regressions, the following tools are recommended:

perf – Linux kernel profiling.

gprof – GNU compiler profiling.

Valgrind – memory and cache analysis.

strace – system‑call tracing.

godbolt.org – inspect generated assembly.

FlameGraph (github.com/brendangregg/FlameGraph) – visualise hot paths.

Each change above was validated with unit tests (e.g., TEST(ParamHitDestructorPerf, test)) and high‑resolution timers to ensure that the measured improvements are reproducible.

Takeaways

Performance tuning is an iterative, data‑driven activity. By profiling hot paths, eliminating heavyweight abstractions (Protobuf), choosing cache‑friendly containers, adopting a thread‑caching allocator, employing lock‑free double buffers for extreme concurrency, and pruning data formats to the minimal required fields, the service achieved measurable latency and memory gains without sacrificing maintainability. Over‑optimisation should be avoided; each optimization must be justified by concrete ROI.