Inside PayPal’s JunoDB: A High‑Performance Distributed KV Store

Introduction

JunoDB is an open‑source distributed key‑value store developed by PayPal to support its global payment platform, which processes roughly 350 billion requests per day. The system targets six‑nines availability (99.9999 %), meaning less than 32 seconds of downtime per year.

Background and Motivation

Existing solutions such as Redis could not meet PayPal’s needs because Redis is single‑threaded and limited by memory, while PayPal required a system whose bottleneck is CPU rather than RAM to handle CPU‑intensive tasks like encryption and complex calculations. Consequently, JunoDB evolved from a single‑threaded C++ prototype for short‑lived data to a highly concurrent, multi‑core‑friendly service rewritten entirely in Go, adding persistence and long‑term storage.

Architecture Overview

JunoDB adopts a three‑layer proxy architecture that separates client applications from physical storage servers, improving scalability, connection management, and operational efficiency. Client requests are first routed to a stateless proxy layer, which then communicates with the appropriate storage server.

Core Components

Client Library

The library is embedded in client applications and offers simple APIs for data storage and retrieval. It is implemented in Java, Go, C++, Node.js, and Python to accommodate PayPal’s diverse technology stack.

Proxy Layer

Connection Pool : Maintains persistent connections to all storage servers, reducing connection overhead as the number of application services grows.

Request Routing : Uses consistent hashing to determine which shard should handle a given key, consulting a shard map for routing.

Configuration Management : Stores shard‑mapping data in etcd, allowing dynamic updates to cluster topology.

Multiple proxy instances run behind a load balancer to avoid a single point of failure.

Storage Server

The stateful layer that executes CRUD operations. It uses RocksDB, Facebook’s high‑performance embedded KV store based on an LSM‑tree, providing high write throughput and supporting both in‑memory and persistent disk storage.

Scalability Techniques

Connection Scaling : Adding more stateless proxy instances allows JunoDB to handle a larger number of inbound connections without performance loss.

Data Scaling : Data is divided into 1 024 fixed shards. Consistent hashing maps keys to shards, and when nodes are added or removed only a minimal set of keys need to be remapped, simplifying cluster expansion and data rebalancing.

Consistency and Availability Mechanisms

Achieving six‑nines availability requires strong fault‑tolerance and data‑consistency strategies.

High Availability : Aggressive replication across multiple physical zones. When a node fails, the system performs instant failover without leader election or data reshuffling. Proxies detect connection loss or timeouts and retry requests on another replica.

Data Consistency : Within a data center, a quorum‑based protocol ensures strong consistency. An operation succeeds only after a majority of replicas agree. The rule is

Write Quorum (W) + Read Quorum (R) > Number of Replicas (N)

. In PayPal’s production environment, N = 5, W = 3, R = 3, guaranteeing that every read sees the latest write. Cross‑region replication is asynchronous to provide disaster recovery without affecting primary latency.

Security Features

Transport Security : All communication among client, proxy, and storage server is encrypted with TLS by default.

At‑Rest Encryption : Data stored on disk is encrypted to prevent unauthorized access.

Key Management : A dedicated module manages certificate and encryption‑key lifecycles, enabling automatic rotation and secure distribution.

PayPal Use Cases

Distributed Cache : Stores frequently accessed but rarely changed data such as user preferences, account details, and API responses, supporting both short‑term (seconds) and long‑term (days) TTL.

Idempotency Store : Holds transaction IDs to ensure that critical operations like payments are not processed multiple times; high availability allows safe retry checks.

Distributed Counter : Central store for tracking resource usage, e.g., API rate limits or authentication attempts.

Latency Bridge : Acts as a fast replication intermediary for legacy dual‑master databases (e.g., Oracle), allowing applications to read the latest data from a secondary JunoDB instance when primary replication lags.

Open‑Source Journey

In 2023 PayPal released JunoDB under the Apache 2.0 license on GitHub, aiming to share the technology with the broader community. The server is written in Go, leveraging its strong concurrency support. The roadmap includes a Kubernetes operator and additional client libraries.

Conclusion

JunoDB demonstrates goal‑driven engineering: identifying unmet requirements—high concurrency, CPU‑bound workloads, and massive scalability—and building a KV store that balances scalability, high availability, strong consistency, and security. It provides a valuable case study for engineers designing large‑scale distributed systems.