Mastering OpenSSL: From SSL/TLS Basics to Practical Encryption Commands

Linux OpenSSL Introduction

SSL and TLS

Before learning OpenSSL, it is essential to understand what SSL/TLS is.

SSL (Secure Sockets Layer) is a protocol that provides an encrypted transport layer for application‑level data, meaning data is encrypted before it moves from the application layer to the transport layer.

As time progressed, its successor TLS (Transport Layer Security) was introduced, offering stronger security, more encryption algorithms, and gradually replacing early SSL versions.

For example: Before SSL/TLS, HTTP requests and responses were plaintext, making them easy to eavesdrop, capture, or tamper with. After introducing SSL/TLS, data is encrypted when moving from the application layer to the transport layer, establishing a secure channel that protects end‑to‑end communication.

Why encrypt application data at the transport layer?

Transport‑layer encryption allows selective encryption per application, whereas network‑layer or data‑link‑layer encryption would encrypt everything.

Transport‑layer encryption ensures end‑to‑end security; lower‑layer encryption may only protect between two devices or networks.

Data encrypted at the transport layer appears unintelligible if captured at the network or data‑link layer.

OpenSSL

OpenSSL is an open‑source toolkit and library for handling SSL/TLS protocols and performing various encryption and decryption operations.

OpenSSL Chinese site: https://www.openssl.net.cn/

The most commonly used tool is the openssl command‑line utility, which can perform tasks such as encryption, decryption, certificate handling, and more.

On most Linux distributions, OpenSSL is pre‑installed; you can check the version with openssl version.

OpenSSL Basic Usage

Implement Symmetric Encryption

Symmetric encryption uses the same key for encryption and decryption, processing data in blocks for high efficiency.

openssl enc -e algorithm_type -a -salt -in <span style="color:#ff7b72">input_file</span> -out <span style="color:#ff7b72">output_file</span>

Example:

openssl enc -e -des3 -a -salt -in file1 -out file1.cipher

Explanation:

Using -salt adds a random salt; OpenSSL generates it automatically.

Besides -des3, other algorithms such as -des are available; see man openssl enc for options.

The -a flag outputs the ciphertext in Base64 format.

Implement Symmetric Decryption

openssl enc -d algorithm_type -a -salt -in <span style="color:#ff7b72">input_file</span> -out <span style="color:#ff7b72">output_file</span>

Example:

openssl enc -d -des3 -a -salt -in file1.cipher -out file1_new

Explanation:

The algorithm type must match the one used for encryption.

Symmetric encryption prompts for a password because the same password is used for both encryption and decryption.

Implement Asymmetric Encryption

Asymmetric encryption uses a key pair: a public key for encryption and a private key for decryption, ensuring data confidentiality and authenticity.

Public‑key encryption + private‑key decryption guarantees that only the holder of the private key can read the data.

Private‑key encryption + public‑key decryption provides digital signatures, confirming the data’s origin.

Generate Private Key

openssl genrsa -out <span style="color:#ff7b72">private_key_file</span> [key_length]

Example:

openssl genrsa -out b.enc

Explanation:

If no key length is specified, the default is 2048 bits; longer keys increase security but also computation time.

The default algorithm is RSA.

The generated key is output in PEM format (Base64‑encoded text).

Generate Public Key

The public key is derived from the private key.

openssl rsa -in <span style="color:#ff7b72">private_key_file</span> -pubout -out <span style="color:#ff7b72">public_key_file</span>

Example:

Implement Asymmetric Decryption

Public‑Key Encryption

openssl rsautl -encrypt -pubin -inkey <span style="color:#ff7b72">public_key_file</span> -in <span style="color:#ff7b72">input_file</span> -out <span style="color:#ff7b72">encrypted_file</span>

Private‑Key Decryption

openssl rsautl -decrypt -inkey <span style="color:#ff7b72">private_key_file</span> -in <span style="color:#ff7b72">encrypted_file</span> -out <span style="color:#ff7b72">decrypted_file</span>

Implement One‑Way Hash

Hash functions produce a fixed‑length digest that cannot be reversed; they are used to verify data integrity.

The concept of a “fingerprint” refers to this unique digest.

openssl dgst <span style="color:#ff7b72">hash_algorithm</span> <span style="color:#ff7b72">file</span>

Example:

openssl dgst file1

Explanation:

If no algorithm is specified, SHA‑256 is used by default; see man openssl dgst for other options.

The default output format is hexadecimal.

Generate User Password

OpenSSL can generate a hashed password suitable for the useradd command.

openssl passwd <span style="color:#ff7b72">algorithm_option</span> <span style="color:#ff7b72">plain_password</span>

If no algorithm is specified, the traditional UNIX crypt() method is used.

Algorithm options: -6: SHA‑512 (adds a random salt). -5: SHA‑256 (adds a random salt). -1: MD5 (adds a random salt).

Example:

useradd bob -m -s /bin/bash -p $(openssl passwd -6 redhat)

Generate Random Number

OpenSSL can output random bytes in hexadecimal or Base64 format.

openssl rand <span style="color:#ff7b72">format</span> <span style="color:#ff7b72">byte_count</span>

Example: generate 16 random bytes in hex:

openssl rand -hex 16

Each hex digit represents 4 bits; therefore 16 hex digits correspond to 8 bytes, and the command outputs 32 characters.

Example: output 16 random bytes in Base64:

openssl rand -base64 16

Base64 Encoding Introduction

Base64 encodes arbitrary binary data using 64 printable ASCII characters. Early protocols like SMTP and HTTP were designed for ASCII only, so Base64 solves the problem of transmitting non‑ASCII or binary data safely.

Encoding Principle

Take three bytes (24 bits) of input and split them into four groups of six bits each.

Convert each 6‑bit group to a decimal value and map it to a character from the Base64 alphabet.

Repeat the process until all input data is processed.

If the input length is not a multiple of three bytes, pad with "=" characters to reach the required length.