Understanding TLS Handshake: Server Certificate, Key Exchange, and Client Authentication

5.4. Handshake — Server Certificate

After the server selects a CipherSuite, if the suite requires authentication, the server sends a Server Certificate message immediately after ServerHello, within the same RTT. The message contains the server's certificate chain.

opaque ASN.1Cert<1..2^24-1>;
struct {
    ASN.1Cert certificate_list<0..2^24-1>;
} Certificate;

The sender's certificate must be first; each subsequent certificate must be signed by the previous one. The root certificate may be omitted.

When a server needs to authenticate the client, it sends a Certificate Request message, and the client replies using the same Server Certificate format.

The server's certificate must:

Be of type X.509v3 unless explicitly negotiated otherwise.

Contain a public key compatible with the chosen key‑exchange algorithm.

Key‑exchange and authentication algorithm compatibility:

RSA / RSA_PSK : RSA public key; certificate must allow keyEncipherment. This usage lacks forward secrecy and is deprecated in TLS 1.3.

DHE_RSA / ECDHE_RSA : RSA public key; certificate must allow digitalSignature and the signature/hash algorithms used in ServerKeyExchange.

DHE_DSS : DSA public key; certificate must allow digitalSignature and the hash algorithms used.

DH_DSS / DH_RSA : Diffie‑Hellman public key; keyAgreement usage bit must be set. This usage lacks forward secrecy and is deprecated in TLS 1.3.

ECDH_ECDSA / ECDH_RSA : ECDH‑capable public key; must use a curve and point format supported by the client. This usage lacks forward secrecy and is deprecated in TLS 1.3.

ECDHE_ECDSA : ECDSA public key; certificate must allow digitalSignature and the chosen hash algorithm.

Extensions such as "server_name" and "trusted_ca_keys" guide certificate selection. ECC key‑exchange algorithms (ECDH_ECDSA, ECDHE_ECDSA, ECDH_RSA, ECDHE_RSA, ECDH_anon) are increasingly important for mobile environments because of smaller keys and higher performance. Only ECDHE_ECDSA and ECDHE_RSA provide forward secrecy.

If the client includes a "signature_algorithms" extension, the server's certificates must be signed with one of the advertised hash/signature pairs. Some certificates use algorithms not supported by TLS (e.g., RSASSA‑PSS) and cannot be used.

5.5. Handshake — Server Key Exchange

The server sends ServerKeyExchange immediately after Server Certificate, or directly after ServerHello for anonymous cipher suites.

ServerKeyExchange is sent only when the Server Certificate does not provide enough information for the client to compute the premaster secret, typically for forward‑secrecy algorithms:

DHE_DSS

DHE_RSA

DH_anon

ECDHE_ECDSA

ECDHE_RSA

ECDH_anon

It is illegal for the following key‑exchange methods to send ServerKeyExchange: RSA, DH_DSS, DH_RSA, ECDH_ECDSA, ECDH_RSA.

The message conveys the server's Diffie‑Hellman or ECDH parameters and, for authenticated exchanges, a digital signature over those parameters.

enum { dhe_dss, dhe_rsa, dh_anon, rsa, dh_dss, dh_rsa, ec_diffie_hellman } KeyExchangeAlgorithm;

struct {
    opaque dh_p<1..2^16-1>;
    opaque dh_g<1..2^16-1>;
    opaque dh_Ys<1..2^16-1>;
} ServerDHParams; // Ephemeral DH parameters

struct { opaque point<1..2^8-1>; } ECPoint;

enum { explicit_prime (1), explicit_char2 (2), named_curve (3), reserved (248..255) } ECCurveType;

struct { ECCurveType curve_type; select (curve_type) { case named_curve: NamedCurve namedcurve; }; } ECParameters;

struct { ECParameters curve_params; ECPoint public; } ServerECDHParams;

struct {
    select (KeyExchangeAlgorithm) {
        case dh_anon: ServerDHParams params;
        case dhe_dss:
        case dhe_rsa: ServerDHParams params; struct { opaque client_random[32]; opaque server_random[32]; ServerDHParams params; } signed_params;
        case ec_diffie_hellman: ServerECDHParams params; Signature signed_params;
        case rsa:
        case dh_dss:
        case dh_rsa: struct {} ;
    };
} ServerKeyExchange;

ECCurveType supports three curve types; in practice, deployments use NamedCurve, with the most common being secp256r1 (P‑256).

enum { secp256r1 (23), secp384r1 (24), secp521r1 (25), reserved (0xFE00..0xFEFF), (0xFFFF) } NamedCurve;

ECDHE_RSA uses RSA as the signature algorithm; ECDHE_ECDSA uses ECDSA.

5.6. Handshake — Certificate Request

The server may request a client certificate after ServerKeyExchange. The message structure is:

enum { rsa_sign(1), dss_sign(2), rsa_fixed_dh(3), dss_fixed_dh(4), rsa_ephemeral_dh_RESERVED(5), dss_ephemeral_dh_RESERVED(6), fortezza_dms_RESERVED(20), ecdsa_sign(64), rsa_fixed_ecdh(65), ecdsa_fixed_ecdh(66), (255) } ClientCertificateType;

opaque DistinguishedName<1..2^16-1>;
struct {
    ClientCertificateType certificate_types<1..2^8-1>;
    SignatureAndHashAlgorithm supported_signature_algorithms<2..2^16-1>;
    DistinguishedName certificate_authorities<0..2^16-1>;
} CertificateRequest;

certificate_types lists the certificate types the client may send (e.g., rsa_sign, dss_sign, rsa_fixed_dh, dss_fixed_dh, ecdsa_sign, rsa_fixed_ecdh, ecdsa_fixed_ecdh). The server also supplies the list of acceptable CAs. If the list is empty, any appropriate certificate may be sent.

Rules:

Any client certificate must be signed with a hash/signature algorithm advertised in supported_signature_algorithms.

The end‑entity certificate must have a key compatible with the requested certificate_type.

If an anonymous cipher suite is negotiated, the server cannot request client authentication.

5.7. Handshake — Server Hello Done

After all server messages are sent, the server transmits ServerHelloDone, indicating that the client may now proceed with its part of the handshake.

struct { } ServerHelloDone;

5.8. Handshake — Client Certificate

The client sends ClientCertificate only if the server requested a certificate. If the client has no suitable certificate, it sends an empty certificate_list.

The server may either continue the handshake or abort with a fatal alert if the certificate is missing or unacceptable.

Client certificates use the same structure as Server Certificate.

Certificates must be X.509v3.

The public key of the leaf certificate must be compatible with the certificate types listed in CertificateRequest.

Supported client certificate types and corresponding public‑key requirements:

rsa_sign – RSA public key; must allow signing with the chosen hash algorithm.

dss_sign – DSA public key; must allow signing with the chosen hash algorithm.

ecdsa_sign – ECDSA public key; must be usable with the server‑supported curve and point format.

rsa_fixed_dh / dss_fixed_dh – Diffie‑Hellman public key; must use the same parameters as the server.

rsa_fixed_ecdh / ecdsa_fixed_ecdh – ECDH public key; must use the same curve and point format as the server.

5.9. Handshake — Client Key Exchange

The client sends ClientKeyExchange immediately after its Certificate (if any) or after ServerHelloDone.

Depending on the key‑exchange algorithm, the message contains either an encrypted premaster secret (RSA) or the client’s Diffie‑Hellman/ECDH public value.

struct {
    select (KeyExchangeAlgorithm) {
        case rsa: EncryptedPreMasterSecret;
        case dhe_dss:
        case dhe_rsa:
        case dh_dss:
        case dh_rsa:
        case dh_anon: ClientDiffieHellmanPublic;
        case ec_diffie_hellman: ClientECDiffieHellmanPublic;
    } exchange_keys;
} ClientKeyExchange;

5.9.(1). RSA‑encrypted Premaster Secret

When RSA is used, the client generates a 48‑byte premaster secret, encrypts it with the server’s RSA public key, and sends the ciphertext.

struct {
    ProtocolVersion client_version;
    opaque random[46];
} PreMasterSecret;

struct { public-key-encrypted PreMasterSecret pre_master_secret; } EncryptedPreMasterSecret;

The client_version field contains the version from ClientHello (not the negotiated version) to prevent version‑rollback attacks. Implementations must handle malformed messages by generating a random premaster secret and continuing the handshake without revealing the error.

5.9.(2). Client Diffie‑Hellman Public Key

The client sends its DH public value Yc, unless it already provided a certificate containing a fixed DH key (implicit case).

enum { implicit, explicit } PublicValueEncoding;

struct {
    select (PublicValueEncoding) {
        case implicit: struct { };
        case explicit: opaque dh_Yc<1..2^16-1>;
    } dh_public;
} ClientDiffieHellmanPublic;

5.9.(3). Client EC Diffie‑Hellman Public Key

struct {
    select (PublicValueEncoding) {
        case implicit: struct { };
        case explicit: ECPoint ecdh_Yc;
    } ecdh_public;
} ClientECDiffieHellmanPublic;

ECDH is the most important key‑exchange algorithm today.

5.10. Handshake — Certificate Verify

If client authentication is required and the client’s certificate can sign, the client sends CertificateVerify to prove possession of the private key. It follows ClientKeyExchange.

struct {
    digitally-signed struct {
        opaque handshake_messages[handshake_messages_length];
    }
} CertificateVerify;

The handshake_messages field contains the concatenation of all prior handshake messages (including type and length fields). The signature must use a hash/signature pair advertised in CertificateRequest and compatible with the client’s certificate.