How Hermes Uses MessagePack for High‑Performance Backend Data Serialization

Introduction

Hermes, named after the Greek messenger god, is a data‑communication protocol designed for the Lest high‑performance, high‑availability active cache system. It leverages the binary MessagePack serialization format to support Java client object serialization and C server‑side object‑tree reconstruction.

Design Goals

Lightweight data exchange

Support for object serialization

Language independence (Java, C, PHP, etc.)

Why Binary Over Text Formats

Although JSON is human‑readable, its verbose metadata reduces transmission efficiency and increases storage size. For large‑scale data access in a cache system, a compact binary format like MessagePack offers higher throughput and lower storage overhead.

MessagePack Protocol Extension

Hermes extends MessagePack with custom type codes to represent objects, maps, lists, and extensions. Key extensions include:

positive fixint : 0xxxxxxx (0x00‑0x7f)

fixmap : 1000xxxx (0x80‑0x8f)

fixarray : 1001xxxx (0x90‑0x9f)

fixstr : 101xxxxx (0xa0‑0xbf)

nil : 11000000 (0xc0)

ext 8/16/32 : 0xc7‑0xc9

int/float types : 0xd0‑0xcb

Custom Ext objects embed a Java class header, a type field, and the serialized binary payload.

Overall Workflow

Data flows in two directions:

Client → Server : Java objects are serialized into a binary stream, transmitted, and deserialized into a C‑side object tree.

Server → Client : The server serializes the object tree back into a binary stream, sends it, and the client reconstructs the original Java objects.

Object Types

Hermes currently supports two container types:

List objects contain key (String), start (int), end (int), and value (ArrayList).

Map objects contain key , start , end , and value (HashMap).

class LestListObject<T>{
    String key;
    int start;
    int end;
    ArrayList<T> value = new ArrayList<>();
}

class LestMapObject<T>{
    String key;
    int start;
    int end;
    Map<K,T> value = new HashMap<>();
}

Client Serialization Example

A LestListObject<String> with key="hello" , start=0 , end=0 , and value=["hello world"] serializes to the following hex stream:

C7 31 00 A3 6B 65 79 A5 68 65 6C 6C 6F A5 73 74 61 72 74 D2 00 00 00 00 A3 65 6E 64 D2 00 00 00 00 A5 76 61 6C 75 65 91 AB 68 65 6C 6C 6F 20 77 6F 72 6C 64

The stream is parsed into five segments, each identified by its MessagePack header (e.g., C7 for an 8‑byte Ext, A3 for a 3‑byte string, D2 for a 32‑bit integer).

Client Deserialization

When the server returns a binary stream, the client scans the stream left‑to‑right, interpreting each header according to its type (INTEGER, LONG, RAW, ARRAY, MAP, NIL, BOOLEAN, FLOAT, DOUBLE, EXT). The supported client‑side types are listed below:

INTEGER: uint8, int8, uint16, int16, uint32, int32

LONG: uint64, int64

RAW: bin8/16/32, str8/16/32

ARRAY: array16, array32

MAP: map16, map32

NIL: nil

BOOLEAN: true/false

FLOAT: float

DOUBLE: double

EXT: ext8, ext16, ext32

Server‑Side Object Tree

Because the server is written in C and lacks native objects, Hermes introduces an object tree where each node ( hms_object ) stores a key, value, type information, and pointers to next (sibling) and child (sub‑node). The structure is:

struct hms_object{
    bool_t is_key;
    object_type key_type;
    object_type value_type;
    struct hermes_object *next;
    struct hermes_object *child;
    object_value value;
    int obj_len;
    int key_len;
    object_value key;
}

Supported data types are enumerated as constants (e.g., HMS_TYPE_STR = 0x00 , HMS_TYPE_INT32 = 0x06 , HMS_TYPE_EXT = 0x11 , etc.). The actual value is stored in a union:

typedef union object_value{
    char *str_val;
    ubyte_t *bin_val;
    int8_t int8_val;
    uint8_t uint8_val;
    int16_t int16_val;
    uint16_t uint16_val;
    int32_t int32_val;
    uint32_t uint32_val;
    int64_t int64_val;
    uint64_t uint64_val;
    float float_val;
    double double_val;
    bool_t bool_val;
} object_value;

Object‑Tree Construction

A binary‑stream scanner parses the incoming data without backtracking. It maintains three pointers: root (tree root), node (current construction node), and buf (data buffer), plus an offset off . For each <header, data> pair, the scanner creates the appropriate hms_object and links it via next or child pointers.

struct scanner{
    struct hms_object *root;
    struct hms_object *node;
    ubyte_t *buf;
    int off;
}

Traversal Strategies

Depth‑first traversal preserves the client’s serialization order but can cause deep recursion when handling long List or Map chains. Hermes mitigates this by using an iterative loop for list/map children while keeping depth‑first semantics for other nodes. In cases where List/Map structures are fixed, breadth‑first traversal yields the same order, allowing a simpler implementation.

Conclusion

Hermes provides a high‑efficiency, language‑agnostic data‑exchange layer for the Lest distributed cache, enabling fast serialization/deserialization, compact binary storage, and flexible query capabilities while reducing network traffic and simplifying server‑side data handling.