TLS Security Deep Dive: Threat Models, Handshakes & Best Practices

9. TLS Protocol Security Analysis

Security analysis is the top priority and the most concerning aspect.

The first step is to define an attacker model: the attacker has abundant computational resources, cannot obtain private keys or in‑memory secrets, but can capture, modify, delete, replay, or tamper with packets. This corresponds to the standard cryptographic attack model.

9.1 Authentication and Key Exchange Security

TLS defines three authentication modes: mutual authentication, server‑only authentication, and no authentication. Any session that includes server authentication is immune to man‑in‑the‑middle attacks; fully anonymous sessions are vulnerable.

The goal of key exchange is to create a shared pre‑master secret, which is used to derive the master secret, the Finished message, the encryption key and the MAC key. Correctly verifying the Finished message proves knowledge of the pre‑master secret.

9.1.1 Anonymous Key Exchange

Anonymous key exchange lacks authentication and is considered insecure; it should be disabled in most deployments.

9.1.2 RSA Key Exchange and Authentication

RSA combines key exchange with server authentication. The RSA public key is contained in the server certificate. If the RSA private key is compromised, all past traffic encrypted with that certificate loses forward secrecy. Users requiring forward secrecy should use DHE or EC‑based cipher suites.

After the server certificate is validated, the client encrypts a pre‑master secret with the server’s public key. Decrypting it and producing a correct Finished message confirms that the server possesses the corresponding private key.

When client authentication is used, the client signs a hash of all previous handshake messages (including the server certificate and ServerHello.random) in a CertificateVerify message, binding the signature to the current handshake.

9.1.3 Diffie‑Hellman Key Exchange and Authentication

For DH key exchange the server either sends a certificate containing fixed DH parameters or sends a temporary DH group signed with the private key of an ECDSA/RSA/DSA certificate. The temporary parameters and hello.random are hashed before signing to prevent replay.

The client validates the certificate or signature to ensure the DH parameters originate from the legitimate server. If the client possesses a certificate with fixed DH parameters, the same DH result will be negotiated each time; therefore the pre‑master secret should be cleared from memory as soon as it is transformed into the master secret.

Implementations must follow RFC 2785 to avoid small‑subgroup attacks, typically by using a DHE cipher suite that generates a fresh DH private key for each handshake.

Clients must verify that the server‑provided DH group size and public exponent meet local policy requirements.

9.2 Version‑Rollback Attacks

Older protocol versions such as SSL 2.0 are vulnerable, so attackers may try to force a connection to downgrade. TLS mitigates this by embedding the protocol version in the PreMasterSecret.

9.3 Handshake‑Manipulation Attacks

Attackers may try to influence the handshake to select weak cipher suites. TLS defends by having both parties compute a hash of all handshake messages; any alteration causes the Finished verification to fail.

9.4 Session‑Resumption Attacks

When resuming a session, new ClientHello.random and ServerHello.random values are hashed together with the master secret. As long as the master secret remains secret and the PRF hash algorithm is secure, the resumed connection is independent of the previous one.

Resumption only proceeds with mutual agreement; either side can abort and request a full handshake if it suspects compromise. Session lifetimes are typically limited to 24 hours, and session IDs should not be stored in persistent storage on insecure hosts.

9.5 Application‑Data Protection Attacks

The master secret, ClientHello.random and ServerHello.random are hashed to derive unique encryption and MAC keys for each connection, even when session resumption is used.

In CBC and stream‑cipher modes, data is MAC‑protected before encryption to prevent replay and tampering. Sequence numbers and distinct write keys for client and server prevent reordering and insertion attacks. If an attacker obtains the encryption key, they can decrypt traffic; if they obtain the MAC key, they can modify messages. AEAD modes simplify this by integrating encryption and authentication.

9.6 Explicit IV Security

TLS 1.0 used the last block of the previous record as the IV for the next record, which enabled the BEAST attack. Modern versions use explicit IVs to mitigate this issue.

9.7 Encryption and MAC Combination Modes

See earlier discussion of CBC versus AEAD; no additional content needed.

9.8 DoS Attack Resilience

Attackers can create many TCP connections to force the server to perform expensive RSA decryption, but random TCP sequence numbers make it difficult to hide the attacker’s IP, allowing standard DoS defenses.

Other DoS vectors such as forged RST packets or malformed TLS records affect any TCP‑based protocol, not just TLS.

9.9 Session Ticket Security Analysis

Tickets must provide both authentication (MAC) and confidentiality (encryption). The TLS specification requires tickets to become invalid when an error is detected, which does not affect their security.

9.9.1 Ticket Theft

An attacker who steals a ticket cannot decrypt it without the server’s ticket‑encryption key, assuming strong algorithms are used.

9.9.2 Ticket Forgery

Strong MAC algorithms such as HMAC‑SHA1 prevent successful forgery or tampering of tickets.

9.9.3 DoS via Tickets

Servers can mitigate ticket‑based DoS by using the key_name field to quickly reject unauthenticated tickets; efficient cryptographic algorithms make this negligible.

9.9.4 Ticket‑Key Management

Keys must be generated with a cryptographically secure random number generator (RFC 4086).

Key length should provide at least 128‑bit security.

Keys are used solely for ticket encryption/decryption.

Keys should be rotated regularly and whenever the ticket format or algorithm changes.

9.9.5 Ticket Lifetime

Servers control ticket lifetime via configuration; lifetimes may exceed 24 hours, but client policies ultimately decide how long tickets are retained.

9.9.6 Alternative Ticket Formats

Non‑standard ticket formats must be carefully analyzed; any confidential data transmitted to the client must be encrypted.

9.9.7 Identity Privacy, Anonymity, and Unlinkability

Encrypted tickets protect sensitive information, but the TLS handshake before ticket decryption can reveal client characteristics, allowing correlation of reused tickets.

10. TLS Extensions

Important extensions include Server Name Indication (SNI), ALPN, Maximum Fragment Length Negotiation, Session Ticket, Safe Renegotiation, and Certificate Status Request (OCSP).

11. PKI Ecosystem

11.1 X.509 Certificates

X.509 defines public‑key certificates, certificate revocation lists (CRL), and certificate‑path validation algorithms. Certificates are serialized using ASN.1 DER and optionally PEM encoding.

Certificate chains consist of one or more CA‑signed certificates ending in a root CA, which is not sent during the TLS handshake.

Certificate fields include public key, usage, validity period, issuer, etc.

11.2 Problems in Existing PKI

Various articles and research papers discuss weaknesses in current PKI implementations and propose mitigations such as public‑key pinning, HSTS, and Certificate Transparency.

12. Historical TLS Vulnerabilities and Cryptographic Pitfalls

12.1 TLS Vulnerabilities

Numerous attacks have been documented, including padding‑oracle, BEAST, Lucky 13, TIME, POODLE, RC4 biases, Heartbleed, and many implementation bugs.

12.2 Common Cryptographic Mistakes

Typical errors include rolling your own crypto, using encryption without MAC, reusing nonces or IVs, using the same key for encryption and authentication, employing ECB mode, and exposing keys derived from passwords.

13. Next‑Generation TLS: TLS 1.3

TLS 1.3 introduces major changes: 0‑RTT and 1‑RTT handshakes, HKDF‑based key derivation, removal of RC4, compression, non‑AEAD algorithms, explicit IVs, ChangeCipherSpec, renegotiation, and static RSA/DH key exchange.

13.1 Record‑Layer Cryptographic Changes

Only AEAD ciphers are permitted, eliminating the need for a separate MAC key. HKDF‑SHA256 or HKDF‑SHA384 are used for key expansion.

13.2 Handshake Changes

Session tickets are integrated as PSK. TLS 1.3 defines three handshake modes: pure DH/ECDH, pure PSK, and a combination of PSK with DH/ECDH.

13.3 1‑RTT Handshake

By fixing the key‑exchange algorithm (typically ECDH P‑256) and caching the server’s choice, the key exchange can be performed in the first RTT, reducing latency.

13.4 0‑RTT Handshake

Clients can send early data using a semi‑static ECDH public key cached from the server, but this sacrifices forward secrecy and is vulnerable to replay attacks.

13.5 Resumption and PSK

TLS 1.3 treats session‑resumption keys and PSK uniformly; PSK‑only handshakes lack forward secrecy, while PSK + ECDHE provides it.

13.6 Key Schedule Changes

Both a static secret (SS) and an ephemeral secret (ES) are derived; the master secret is computed from both. Early‑data keys use SS (no forward secrecy), handshake keys use ES, and application keys use the combined master secret.

struct {
    opaque configuration_id<1..2^16-1>;
    uint32 expiration_date;
    NamedGroup group;
    opaque server_key<1..2^16-1>;
    EarlyDataType early_data_type;
    ConfigurationExtension extensions<0..2^16-1>;
} ServerConfiguration;