Understanding SSL/TLS and DTLS: From Cryptographic Basics to WebRTC Security

0. Background

Early Internet protocols were designed without considering data‑transmission security, exposing three main risks: eavesdropping, tampering, and impersonation.

To mitigate these risks, Netscape created SSL in 1994, later standardized as TLS (first version in 1999, with TLS 1.3 as the latest). SSL/TLS prevents eavesdropping by encrypting all data, detects tampering through integrity checks, and stops impersonation with certificates.

1. Cryptographic Foundations

Symmetric Key Encryption

Symmetric encryption uses the same key for encryption and decryption. It includes block ciphers (DES, 3DES, AES, Blowfish, IDEA) and stream ciphers (RC6, ChaCha20‑Poly1305). Stream ciphers cannot be used in DTLS.

Asymmetric Key Encryption

Asymmetric encryption uses a public key and a private key. Common algorithms are RSA, DSA, and Diffie‑Hellman. It avoids key‑distribution problems but is slower and CPU‑intensive.

Hash (Message Digest)

Hash functions compress arbitrary‑length input to a fixed‑length output (e.g., MD5, SHA‑1, SHA‑256). They are one‑way functions and are used for data integrity.

Message Authentication Code (MAC)

MAC provides integrity and authentication of a message, similar to a keyed hash, and can detect any modification.

Digital Signature

A digital signature encrypts a hash of the message with the sender’s private key; the receiver verifies it with the public key, ensuring authenticity and non‑repudiation.

Certificate Authority (CA)

CA issues digital certificates that bind a public key to an identity, preventing certificate forgery.

Digital Certificate

A digital certificate contains identity information, the public key, and the CA’s signature, solving the public‑key distribution problem.

2. AES Symmetric Encryption Algorithm

AES is the most widely used block cipher. It operates on 128‑bit blocks with key lengths of 128, 192, or 256 bits and supports several modes: ECB, CBC, CTR, GCM, etc.

2.1 Padding

Supported paddings: NoPadding, PKCS#7, ISO 10126, ANSI X9.23, ZerosPadding.

NoPadding means no padding; it only works when the plaintext length is a multiple of 128 bits.

PKCS#7 pads the last block with the number of padding bytes (e.g., 04 04 04 04).

ISO 10126 pads with random bytes and ends with the length byte.

ANSI X9.23 pads with zeros except for the final length byte.

ZerosPadding fills the remaining bytes with 0x00; it requires that the original plaintext does not contain 0x00.

2.2 Electronic Codebook (ECB)

ECB encrypts each block independently, which reveals patterns when the same plaintext block repeats, making it insecure for most data.

2.3 Cipher Block Chaining (CBC)

In CBC, each plaintext block is XORed with the previous ciphertext block before encryption, using an IV for the first block.

CBC provides confidentiality but is sequential, limiting parallelism; decryption can be parallel.

2.4 Counter (CTR)

CTR mode generates a keystream by encrypting a nonce concatenated with a counter; it allows parallel encryption and does not require padding.

2.5 GCM (CTR + GMAC)

GCM combines CTR mode with a Galois Message Authentication Code, providing both confidentiality and integrity.

2.6 AES Algorithm Steps

The number of rounds depends on the key length:

AES

Key Length (words)

Block Length (words)

Rounds

AES‑128

4

4

10

AES‑192

6

4

12

AES‑256

8

4

14

The encryption formula is C = E(K, P). Each round consists of SubBytes, ShiftRows, MixColumns (except the final round), and AddRoundKey.

2.6.1 Key Expansion

The 128‑bit key is expanded into 44 32‑bit words (W[0]‑W[43]); the first four words form the initial round key, and each subsequent group of four words is used for the next round.

2.6.2 Overall Process

The AES flow starts with an initial AddRoundKey, followed by nine identical rounds (SubBytes, ShiftRows, MixColumns, AddRoundKey) and a final round without MixColumns.

SubBytes

A fixed 16×16 S‑box substitutes each byte based on its high and low nibble.

ShiftRows

Rows are cyclically shifted left by 0, 1, 2, 3 bytes respectively.

MixColumns

Each column is multiplied by a fixed matrix in GF(2⁸), providing diffusion.

3. TLS

DTLS extends TLS to work over UDP, adding fields for ordering and retransmission.

SSL/TLS uses public‑key cryptography to exchange a symmetric session key, then encrypts data with that key. Two main issues are ensuring the public key is authentic (solved by certificates) and reducing the computational cost of public‑key operations (solved by using the session key for bulk encryption).

The TLS handshake consists of a client‑hello, server‑hello, certificate exchange, key exchange, and optional certificate verification, followed by a change‑cipher‑spec and encrypted handshake messages.

3.1 Record Layer

<span>struct {</span>
  <span>uint8 major;</span>
  <span>uint8 minor;</span>
<span>} ProtocolVersion;</span>

ProtocolVersion tls_version = { 3, 3 };          /* TLS 1.2 */
ProtocolVersion dtls_version = { 254, 253 };     /* DTLS 1.2 */

enum { change_cipher_spec(20), alert(21), handshake(22), application_data(23), (255) } ContentType;

struct { ContentType type; ProtocolVersion version; uint16 length; opaque fragment[TLSPlaintext.length]; } TLSPlaintext;

struct { ContentType type; ProtocolVersion version; uint16 epoch; uint48 sequence_number; uint16 length; opaque fragment[DTLSPlaintext.length]; } DTLSPlaintext;

struct { ContentType type; ProtocolVersion version; uint16 length; opaque fragment[TLSCompressed.length]; } TLSCompressed;

struct { ContentType type; ProtocolVersion version; uint16 length; select (SecurityParameters.cipher_type) { case stream: GenericStreamCipher; case block: GenericBlockCipher; case aead: GenericAEADCipher; } fragment; } TLSCiphertext;

3.2 Handshake

<span>enum { hello_request(0), client_hello(1), server_hello(2), hello_verify_request(3), certificate(11), server_key_exchange(12), certificate_request(13), server_hello_done(14), certificate_verify(15), client_key_exchange(16), finished(20), (255) } HandshakeType;</span>

struct { HandshakeType msg_type; uint24 length; select (HandshakeType) { case hello_request: HelloRequest; case client_hello: ClientHello; case server_hello: ServerHello; case certificate: Certificate; case server_key_exchange: ServerKeyExchange; case certificate_request: CertificateRequest; case server_hello_done: ServerHelloDone; case certificate_verify: CertificateVerify; case client_key_exchange: ClientKeyExchange; case finished: Finished; } body; } Handshake;

3.2.1 ClientHello

Contains the supported protocol versions (ordered by preference), a 32‑byte random value, an optional session ID, and a list of supported cipher suites.

3.2.2 ServerHello

The server selects a protocol version, random value, session ID, and a cipher suite from the client’s list.

3.2.3 Certificate, Server Key Exchange, ServerHelloDone

The server sends its certificate chain, optionally a ServerKeyExchange (required for algorithms like ECDHE), and a ServerHelloDone message.

3.2.4 Client Key Exchange, Change Cipher Spec, Encrypted Handshake Message

The client sends a ClientKeyExchange (encrypted pre‑master secret for RSA or DH parameters for ECDHE), then a ChangeCipherSpec, followed by an encrypted Finished message.

4. DTLS

DTLS adapts TLS for unreliable UDP transport by adding fields for epoch, sequence numbers, and fragment handling to guarantee ordered delivery and retransmission.

4.1 Handshake Fields

struct { ContentType type; ProtocolVersion version; uint16 epoch; uint48 sequence_number; uint16 length; opaque fragment[DTLSPlaintext.length]; } DTLSPlaintext;

struct { HandshakeType msg_type; uint24 length; uint16 message_seq; uint24 fragment_offset; uint24 fragment_length; select (HandshakeType) { case hello_request: HelloRequest; case client_hello: ClientHello; case server_hello: ServerHello; case hello_verify_request: HelloVerifyRequest; case certificate: Certificate; case server_key_exchange: ServerKeyExchange; case certificate_request: CertificateRequest; case server_hello_done: ServerHelloDone; case certificate_verify: CertificateVerify; case client_key_exchange: ClientKeyExchange; case finished: Finished; } body; } Handshake;

4.2 ClientHello (same as TLS)

Identical to the TLS ClientHello, but includes an optional cookie for DoS protection.

4.3 HelloVerifyRequest

If the server wants to mitigate DoS attacks, it replies to the first ClientHello with a HelloVerifyRequest containing a cookie; the client must resend ClientHello with that cookie.

4.4 Server Hello, Certificate, Server Key Exchange, Certificate Request, ServerHelloDone

These messages are identical to the TLS equivalents.

4.5 New Session Ticket, Change Cipher Spec, Encrypted Handshake Message

DTLS reuses the TLS mechanisms for session resumption.

5. DTLS in WebRTC

DTLS is used in WebRTC for encrypting the data channel and for negotiating SRTP keys. WebRTC typically employs self‑signed certificates, so the fingerprint of the certificate is exchanged over the signaling channel to verify authenticity before the DTLS handshake.

Key attributes: a=setup: legal values are actpass, active, or passive. The offerer usually sets actpass, and the answerer selects active or passive. a=fingerprint: the hash of the certificate (e.g., sha-256) is exchanged to detect tampering.