How Adding a Distributed Memcache Layer Cut Database Load by 30% in a Telecom CRM

Background

The core CRM system of a telecom operator experienced rapidly increasing OLTP load, especially on frequently accessed tables such as dictionary and three‑party customer data. Traditional scaling methods—hardware expansion and Oracle RAC—could not keep up with demand and incurred high costs.

Solution Architecture

A horizontally scalable distributed cache layer based on Memcached was added on x86 servers. Frequently accessed tables were moved into the cache and the application was refactored to read and write through Memcached, dramatically reducing database hits.

Memcached Overview

Memcached is a high‑performance distributed in‑memory key/value store written in C, with client libraries for many languages. It reduces database load by caching objects in memory. The project used version 1.4.

Application Access Logic

When a session accesses a table, first check the cache; if present, return the data, otherwise read from the database and write the result into the cache.

If multiple cache nodes exist, a front‑end dispatcher routes the request to a specific node based on a hash rule.

If a cache miss or cache failure occurs, the application falls back to the database without affecting the user experience.

On data updates, the database is updated first, then the cache is refreshed synchronously.

Cache updates propagate to all other cache nodes; the overhead is negligible because the operation occurs in memory.

High‑Availability Measures

If a Memcached server becomes unavailable, the dispatcher receives an error, triggers a direct database query, and, if multiple clusters exist, may query another cluster. When the failed server recovers, normal cache routing resumes.

Performance Results

After caching the top 30 most‑accessed tables (covering the top 10 business functions that account for >80% of traffic), overall database accesses dropped by 30% and key business response times improved by 15%.

Database pressure was reduced by 40%, eliminating an anticipated 15% growth and saving roughly 20 million CNY in expansion costs.

Implementation Highlights

Cache adoption required substantial application refactoring; all reads now go through a key/value lookup.

Read‑heavy, write‑light tables (read/write ratio >8:2) are ideal for caching; write‑heavy tables see limited benefit.

The Memcached cluster consists of 40 x86 nodes (≈30 GB RAM each) organized into four shards.

Cache consistency is maintained by updating the database first, then synchronously refreshing the cache via application logic.

Redis was considered but Memcached was chosen due to existing expertise.

Sample Cache Access Code

function get_data($key) {
    $data = $this->cache->get($key);
    if ($data !== null) {
        return $data;
    } else {
        if ($this->cache->getResultCode() == Memcached::RES_NOTFOUND) {
            // Query the database
            $data = /* result from DB */;
            $this->cache->set($key, $data);
            return $data;
        } else {
            error_log('no data for key ' . $key);
        }
    }
}