Understanding TLS Record Protocol: Encryption, MAC, and Key Management

4. record protocol

Record protocol performs symmetric encryption of application data and occupies the majority of traffic on a TLS connection, so it is essential to understand its operation.

Record protocol – from the application layer it receives data and then:

Fragments the data (reassembly is the reverse process).

Generates a sequence number for each fragment to prevent replay or reordering.

Optionally compresses the fragment using the algorithm negotiated in the handshake.

Encrypts the fragment with the key negotiated in the handshake.

Computes an HMAC over the data and verifies the received packet's HMAC.

Sends the resulting packet to TCP/IP (or another IPC mechanism) for transmission.

4.1. SecurityParameters

The processing above is driven entirely by the parameters defined in SecurityParameters:

struct {
    ConnectionEnd  entity;
    PRFAlgorithm   prf_algorithm;
    BulkCipherAlgorithm    bulk_cipher_algorithm;
    CipherType     cipher_type;
    uint8          enc_key_length;
    uint8          block_length;
    uint8          fixed_iv_length;
    uint8          record_iv_length;
    MACAlgorithm   mac_algorithm;
    uint8          mac_length;
    uint8          mac_key_length;
    CompressionMethod  compression_algorithm;
    opaque         master_secret[48];
    opaque         client_random[32];
    opaque         server_random[32];
} SecurityParameters;

From these parameters the record layer derives six values (client/server MAC keys, encryption keys, and IVs). After the handshake completes, these values become part of the connection state, which also includes compression state, cipher state, and a 64‑bit sequence number that must start at zero and never wrap.

4.2. Record Layer Fragmentation

Data to be sent is first placed into a TLSPlaintext structure:

struct {
    ContentType type;
    ProtocolVersion version;
    uint16 length;
    opaque fragment[TLSPlaintext.length];
} TLSPlaintext;

version : the negotiated TLS version (e.g., TLS 1.2 is {3,3}).

length : must be less than 2^14 to avoid excessive latency.

type : identifies whether the record contains a handshake, alert, application data, etc.

Compression is optional but should be disabled in practice because of attacks such as CRIME and BREACH.

4.3. Cryptographic Protection in the Record Layer

After processing, the record is formatted as TLSCiphertext:

struct {
    ContentType type;
    ProtocolVersion version;
    uint16 length;
    select (SecurityParameters.cipher_type) {
        case stream: GenericStreamCipher;
        case block:  GenericBlockCipher;
        case aead:   GenericAEADCipher;
    } fragment;
} TLSCiphertext;

The TLS design goals of confidentiality and integrity are achieved via three families of constructions:

Block cipher (CBC) + HMAC (e.g., aes‑128‑cbc + hmac‑sha256).

Stream cipher (RC4) + HMAC.

Authenticated‑Encryption (AEAD) such as aes‑gcm.

Modern TLS implementations favor AEAD because it avoids the pitfalls of MAC‑then‑Encrypt, which led to attacks like BEAST, Lucky 13, and POODLE.

4.4. MAC Calculation

MAC(MAC_write_key, seq_num +
    TLSCompressed.type +
    TLSCompressed.version +
    TLSCompressed.length +
    TLSCompressed.fragment);

The sequence number and header fields are included to prevent replay attacks and ensure integrity of the entire record.

4.5. Stream Cipher Protection

Historically RC4 was used, but it is now considered insecure and is no longer recommended.

4.6. CBC Block Cipher Protection

Typical block ciphers are AES, Camellia, and SEED. The record format includes an IV, the encrypted content, MAC, padding, and padding length. The IV must be generated with a cryptographically secure RNG. TLS 1.2 introduced an explicit IV field to avoid the BEAST attack.

4.7. AEAD Cipher Protection

AEAD modes (aes‑gcm, chacha20‑poly1305) combine encryption and authentication. The record format is:

struct {
    opaque nonce_explicit[SecurityParameters.record_iv_length];
    aead-ciphered struct {
        opaque content[TLSCompressed.length];
    };
} GenericAEADCipher;

AEAD input consists of the write key, a nonce derived from the sequence number and static IV, the plaintext, and additional data (seq_num, type, version, length). Decryption fails with a fatal alert if integrity verification does not succeed.

4.8. Key Expansion

The master secret (48 bytes) is expanded using the PRF to produce the key block, which is then split into MAC keys, encryption keys, and IVs for both client and server:

key_block = PRF(SecurityParameters.master_secret, "key expansion",
    SecurityParameters.server_random + SecurityParameters.client_random);

client_write_MAC_key[SecurityParameters.mac_key_length]
server_write_MAC_key[SecurityParameters.mac_key_length]
client_write_key[SecurityParameters.enc_key_length]
server_write_key[SecurityParameters.enc_key_length]
client_write_IV[SecurityParameters.fixed_iv_length]
server_write_IV[SecurityParameters.fixed_iv_length]

TLS 1.3 replaces this PRF with HKDF for stronger key derivation.