Understanding TLS Handshake: Protocol Flow, Algorithms, and Security

5. Handshake Protocol

The handshake protocol is essential and intricate. TLS 1.3 introduced major changes, but this article first explains TLS 1.2 before briefly touching on TLS 1.3.

5.1 Overall Handshake Flow

The handshake protocol creates the SecurityParameters used by the record protocol. During the handshake the client and server:

negotiate the TLS version and a CipherSuite,

optionally authenticate each other's identity (e.g., HTTPS authenticates the server),

use a key‑exchange algorithm to generate a shared master secret.

The steps are:

Exchange Hello messages to agree on algorithms, exchange random values, and check for session resumption.

Exchange necessary cryptographic parameters to allow client and server to derive a premaster secret.

Exchange certificates and cryptographic parameters so both sides can authenticate each other.

Derive the master secret from the premaster secret and the exchanged random values.

Provide the SecurityParameters to the record layer.

Confirm that both sides derived identical SecurityParameters and that the handshake data was not tampered with.

The handshake result is a shared Session containing:

session identifier – uniquely identifies a session and is used for session resumption,

peer certificate – the X.509v3 certificate of the opposite party (empty if authentication is not required),

compression method – usually disabled,

cipher spec – the chosen CipherSuite (PRF, block cipher, MAC, etc.),

master secret – a 48‑byte shared key,

is resumable – flag indicating whether the session can be resumed.

These fields are later used to generate the record‑layer SecurityParameters, and multiple connections can reuse the same session via the session‑resumption feature.

Asymmetric Algorithm Categories

Asymmetric encryption: RSAES‑PKCS1‑v1_5, RSAES‑OAEP, Rabin‑Williams‑OAEP, Rabin‑Williams‑PKCS1‑v1_5, etc.

Asymmetric key‑exchange: DH, DHE, ECDH, ECDHE, etc.

Asymmetric digital signatures: RSASSA‑PKCS1‑v1_5, RSASSA‑PSS, ECDSA, DSA, ED25519, etc.

Note: RSA encryption schemes (RSAES‑PKCS1‑v1_5, RSAES‑OAEP) can also be used as key‑exchange algorithms.

RSA Details

Practical RSA implementations follow the PKCS#1 standard ( https://www.ietf.org/rfc/rfc3447 ). RSAES‑PKCS1‑v1_5 and RSASSA‑PKCS1‑v1_5 are two different schemes: RSAES for encryption, RSASSA for signatures. RSAES‑OAEP is the more secure encryption scheme; RSASSA‑PSS is the more secure signature scheme.

A generally good cryptographic practice is to employ a given RSA key pair in only one scheme. This avoids the risk that a vulnerability in one scheme may compromise the security of the other, and may be essential to maintain provable security.

An RSA key pair used for digital signatures shall only be used for one digital signature scheme (e.g., ANS X9.31, RSASSA‑PKCS1 v1.5 or RSASSA‑PSS). In addition, an RSA digital signature key pair shall not be used for other purposes (e.g., key establishment).

Key pairs must be used for a single purpose only – either encryption/decryption or signing/verification, never both.

This principle underpins forward secrecy (PFS). After the Snowden revelations, PFS became especially important.

Downgrade Attacks

Older algorithms from the 1990s (e.g., RC4, short‑key RSA) are now broken. Because TLS must remain backward compatible, implementations still contain code for these weak algorithms. An attacker who forces a handshake to use such an algorithm can compromise security – a "downgrade attack".

To mitigate downgrade attacks, TLS requires the client to send a ClientHello and the server to reply with a ServerHello. If the server does not respond, the connection is terminated as a fatal error.

Protocol Version

Session ID

Cipher Suite

Compression Method

Both parties also exchange ClientHello.random and ServerHello.random values.

Key‑Exchange Message Flow

The four messages used for key exchange are Certificate, ServerKeyExchange, Certificate (client), and ClientKeyExchange. The shared secret must be sufficiently long (current algorithms generate keys > 46 bytes).

Client                                 Server

ClientHello          --------->
                     <------- ServerHello
                     Certificate*
                     ServerKeyExchange*
                     CertificateRequest*
                     <------- ServerHelloDone
Certificate*          
ClientKeyExchange
CertificateVerify*
[ChangeCipherSpec]
Finished               --------->
                     [ChangeCipherSpec]
                     <------- Finished
Application Data       <-------> Application Data

* optional messages

Note: ChangeCipherSpec is a separate TLS content type, not a handshake message.

Performance of Asymmetric Operations (2015)

On an Intel Xeon E3‑1230 V2 (3.30 GHz) the following benchmark results were observed:

RSA‑2048: private‑key 723.7 ops/s, public‑key 23 505.8 ops/s

ECDSA‑P‑256: sign 8 628.4 ops/s, verify 2 217.0 ops/s

ECDH‑P‑256: key‑exchange 2 807.8 ops/s

AES‑128‑CBC: encryption 121 531.39 KB/s (without hardware acceleration)

AES‑128‑CBC with AES‑NI: encryption 683 682.13 KB/s

Asymmetric operations are orders of magnitude slower than symmetric ones, which motivates TLS to cache handshake results via session IDs or session tickets.

Session Resumption (Cache)

When a client presents a known Session ID in ClientHello, the server looks it up in its cache. If found, the server returns the same Session ID in ServerHello, both sides send ChangeCipherSpec and Finished, and the connection proceeds without a full handshake. If not found, a new Session ID is generated.

Client                                 Server

ClientHello          --------->
                     <------- ServerHello
                     [ChangeCipherSpec]
                     <------- Finished
[ChangeCipherSpec]
Finished               --------->
Application Data       <-------> Application Data

5.2 Handshake Protocol Outer Structure

The TLS Handshake Protocol sits above the TLS Record Protocol. Handshake messages (e.g., ClientHello, ServerHello) are wrapped in a TLSPlaintext structure and processed according to the current session state.

enum {
    hello_request(0), client_hello(1), server_hello(2),
    certificate(11), server_key_exchange(12),
    certificate_request(13), server_hello_done(14),
    certificate_verify(15), client_key_exchange(16),
    finished(20), (255)
} HandshakeType;

struct {
    HandshakeType msg_type;  // handshake type
    uint24 length;           // bytes in message
    select (HandshakeType) {
        case hello_request:       HelloRequest;
        case client_hello:        ClientHello;
        case server_hello:        ServerHello;
        case certificate:        Certificate;
        case server_key_exchange:ServerKeyExchange;
        case certificate_request:CertificateRequest;
        case server_hello_done:  ServerHelloDone;
        case certificate_verify: CertificateVerify;
        case client_key_exchange:ClientKeyExchange;
        case finished:           Finished;
        case session_ticket:    NewSessionTicket; // NEW
    } body;
} Handshake;

Handshake messages must be sent in the defined order; out‑of‑order messages trigger a fatal error.

5.3 Handshake – ClientHello, ServerHello, HelloRequest

5.3.1 ClientHello

The first message a client sends on a new connection is ClientHello. If the server later sends a HelloRequest, the client may send another ClientHello to renegotiate.

struct {
    uint32 gmt_unix_time;
    opaque random_bytes[28];
} Random;

opaque SessionID<0..32>;
uint8 CipherSuite[2];

enum { null(0), (255) } CompressionMethod;

struct {
    ProtocolVersion client_version;
    Random random;
    SessionID session_id;
    CipherSuite cipher_suites<2..2^16-2>;
    CompressionMethod compression_methods<1..2^8-1>;
    select (extensions_present) {
        case false: struct {};
        case true:  Extension extensions<0..2^16-1>;
    };
} ClientHello;

The random field consists of a Unix timestamp and 28 bytes of cryptographically secure random data. Modern implementations should use /dev/urandom or RAND_bytes(). Historical bugs (e.g., early Netscape) used predictable seeds and should be avoided.

5.3.2 ServerHello

struct {
    ProtocolVersion server_version;
    Random random;
    SessionID session_id;
    CipherSuite cipher_suite;
    CompressionMethod compression_method;
    select (extensions_present) {
        case false: struct {};
        case true:  Extension extensions<0..2^16-1>;
    };
} ServerHello;

The server selects the lowest version supported by both sides, generates its own random, and either reuses the client's Session ID (session resumption) or issues a new one. The chosen CipherSuite must match the resumed session if resumption occurs.

5.3.3 Hello Extensions

struct {
    ExtensionType extension_type;
    opaque extension_data<0..2^16-1>;
} Extension;

enum { signature_algorithms(13), (65535) } ExtensionType;

Extensions are optional and must appear at most once per type. Common extensions include SNI, session tickets, ALPN, and OCSP.

5.3.4 HelloRequest

struct { } HelloRequest;

The server may send HelloRequest at any time to prompt the client to start a new handshake. The client can ignore it; if it does not respond, the server may abort with a fatal alert.