How Pipes, Named Pipes, and Message Queues Revolutionized Process Communication

In the 1970s, Bell Labs system engineers introduced process isolation, giving each process its own address space and preventing interference. However, isolated processes lacked an efficient way to exchange data.

Initially, data exchange was simulated using magnetic tape: three processes read and write sequentially to the tape, performing format conversion, analysis, and reporting. This approach suffered from two major drawbacks:

Every communication required a full tape I/O operation, making it extremely slow.

Processes could not communicate in real time; each had to wait for the previous one to finish.

To overcome these limits, engineers conceived the first breakthrough IPC mechanism: the pipe.

Pipe – The First Breakthrough IPC Mechanism

Instead of using tape, a pipe creates an in‑memory FIFO queue where one process writes data and another reads it. A simple prototype used the following structure:

struct pipe {
    char buffer[PIPE_SIZE];   // circular buffer
    int read_pos;            // read index
    int write_pos;           // write index
    struct proc *reader;      // reading process
    struct proc *writer;      // writing process
};

This design solved key problems:

Data is transferred directly in memory, eliminating slow tape I/O.

By using file descriptors, parent and child processes can share the pipe, though this restricts usage to related processes.

While pipes quickly became popular and formed the basis of IPC in operating systems, their limitations soon appeared:

Only processes with a parent‑child relationship can use them.

Pipes are unidirectional; bidirectional communication requires two pipes.

The data stream has no inherent message boundaries, making it a raw byte stream.

Named Pipe – Removing the Parent‑Child Restriction

To allow any two processes to communicate, the pipe concept was extended with a name registered in the file system. Any process could open the named pipe via its path, turning the pipe into a file‑like object.

The resulting named pipe (FIFO) offered several advantages:

It works between unrelated processes, breaking the parent‑child limitation.

It leverages the existing file‑system permission model.

Programmers can use standard file I/O calls, simplifying development.

Message Queue – Structured Communication

As applications grew more complex, named pipes showed shortcomings: they remained unidirectional, lacked explicit message boundaries, and offered no built‑in multi‑client support. To address these issues, a new IPC mechanism—message queues—was designed.

Message queues provide structured messages with clear boundaries, priority handling, and type‑based retrieval. A simplified implementation uses the following structures:

struct msg_msg {
    struct list_head m_list;   // linked‑list pointer
    long m_type;               // message type for classification
    size_t m_ts;                // total size (data + metadata)
    char m_data[0];             // flexible array for payload
};

struct msg_queue {
    struct list_head q_messages; // head of message list
    unsigned long q_cbytes;      // total bytes in queue
    unsigned long q_qnum;        // number of messages in queue
};

Message queues dramatically simplify communication in systems that need to handle multiple message types and priorities.

Despite their benefits, these IPC mechanisms share common performance challenges:

Data must be copied from user space to kernel buffers and back, incurring CPU and memory overhead.

High‑throughput workloads (e.g., image processing, scientific computing, databases) demand lower latency and higher bandwidth than traditional IPC can provide.

Complex data structures require serialization before transmission, adding development complexity and performance cost.

These limitations motivated further research into more efficient IPC techniques, such as shared memory, zero‑copy networking, and advanced kernel bypass methods.